{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import sys\n",
    "# sys.path # python导入package时自动搜索的路径\n",
    "# sys.path.append(\"/root/Docqa\")\n",
    "import sys\n",
    "sys.path\n",
    "sys.path.append(\"/root/\")  \n",
    "from langchain.text_splitter import CharacterTextSplitter,RecursiveCharacterTextSplitter\n",
    "from langchain.document_loaders import Docx2txtLoader\n",
    "from langchain.document_loaders import PyPDFLoader, ArxivLoader,PyPDFDirectoryLoader\n",
    "from Docqa.text_spliter import ChineseTextSplitter\n",
    "import PyPDF2 # pdf reader\n",
    "from pypdf import PdfReader\n",
    "from langchain.chains import RetrievalQA\n",
    "from langchain.memory import ConversationBufferMemory\n",
    "# from langchain.chains import ConversationalRetrievalChain\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "loader = Docx2txtLoader('/data/documents/分布式计算系统.docx')\n",
    "input_doc = loader.load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=650, chunk_overlap=0)\n",
    "texts = text_splitter.split_documents(input_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "loader = PyPDFLoader(\"/data/documents/深度学习_582660_231113.pdf\")\n",
    "pages = loader.load()\n",
    "pdf_splitter = ChineseTextSplitter(chunk_size=650, chunk_overlap=0, pdf=True)\n",
    "pdf_text = pdf_splitter.split_documents(pages)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = PyPDFLoader(\"/data/documents/2304.04912.pdf\")\n",
    "pages = loader.load()\n",
    "pdf_splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=165)\n",
    "pdf_text = pdf_splitter.split_documents(pages)\n",
    "len(pdf_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Financial Time Series Forecasting using CNN and Transformer\\nZhen Zeng, Rachneet Kaur, Suchetha Siddagangappa,\\nSaba Rahimi, Tucker Balch, Manuela Veloso\\nJ. P. Morgan AI Research, New York, NY , USA\\nAbstract\\nTime series forecasting is important across various domains\\nfor decision-making. In particular, ﬁnancial time series such\\nas stock prices can be hard to predict as it is difﬁcult to\\nmodel short-term and long-term temporal dependencies be-\\ntween data points. Convolutional Neural Networks (CNN) are\\ngood at capturing local patterns for modeling short-term de-\\npendencies. However, CNNs cannot learn long-term depen-\\ndencies due to the limited receptive ﬁeld. Transformers on the\\nother hand are capable of learning global context and long-\\nterm dependencies. In this paper, we propose to harness the\\npower of CNNs and Transformers to model both short-term\\nand long-term dependencies within a time series, and forecast\\nif the price would go up, down or remain the same (ﬂat) in', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0}),\n",
       " Document(page_content='and long-term dependencies within a time series, and forecast\\nif the price would go up, down or remain the same (ﬂat) in\\nthe future. In our experiments, we demonstrated the success\\nof the proposed method in comparison to commonly adopted\\nstatistical and deep learning methods on forecasting intraday\\nstock price change of S&P 500 constituents.\\nIntroduction\\nTime series forecasting is challenging, especially in the ﬁ-\\nnancial industry (Pedersen 2019). It involves statistically\\nunderstanding complex linear and non-linear interactions\\nwithin historical data to predict the future. In the ﬁnan-\\ncial industry, common applications for forecasting include\\npredicting buy/sell or positive/negative price changes for\\ncompany stocks traded on the market. Traditional statis-\\ntical approaches commonly adapt linear regression, expo-\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\nand McKenzie 1985) and autoregression models (Makri-\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0}),\n",
       " Document(page_content='nential smoothing (Holt 2004; Winters 1960; Gardner Jr\\nand McKenzie 1985) and autoregression models (Makri-\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-\\nvances in deep learning, recent works are heavily invested in\\nensemble models and sequence-to-sequence modeling such\\nas Recurrent Neural Networks (RNNs), Long Short-Term\\nMemory (LSTM) (Hochreiter and Schmidhuber 1997). In\\nComputer Vision domain, Convolutional Neural Networks\\n(CNN) (Ren et al. 2015; Ronneberger, Fischer, and Brox\\n2015) have shown prominence in learning local patterns\\nwhich are suitable for modeling short-term dependencies,\\nalthough not suitable for modeling long-term dependencies\\ndue to limited receptive ﬁeld. Most recently, Transform-\\ners (Vaswani et al. 2017), have shown great success in Nat-\\nCopyright © 2023, Association for the Advancement of Artiﬁcial\\nIntelligence (www.aaai.org). All rights reserved.ural Language Processing (NLP) (Devlin et al. 2018; Brown\\net al. 2020; Smith et al. 2022) domain, achieving supe-', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0}),\n",
       " Document(page_content='Intelligence (www.aaai.org). All rights reserved.ural Language Processing (NLP) (Devlin et al. 2018; Brown\\net al. 2020; Smith et al. 2022) domain, achieving supe-\\nrior performance on long-term dependencies modeling com-\\npared to LSTM.\\nOur contributions is the following: we leverage the advan-\\ntages of CNNs and Transformers to model short-term and\\nlong-term dependencies in ﬁnancial time series, as shown\\nin Figure 1. In our experiments, we show the advantage of\\nthe proposed approach on intraday stock price prediction\\nof S&P 500 constituents, outperforming statistical meth-\\nods including Autoregressive Integrated Moving Average\\n(ARIMA) and Exponential Moving Average (EMA) and\\na state-of-the-art deep learning-based autoregressive model\\nDeepAR (Salinas et al. 2020).\\nRelated Works\\nTime series forecasting\\nTypical forecasting techniques in the literature utilize sta-\\ntistical tools, such as, exponential smoothing (ETS) (Holt\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0}),\n",
       " Document(page_content='tistical tools, such as, exponential smoothing (ETS) (Holt\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\nand autoregressive integrated moving average (ARIMA)\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\nmerical time series data for making one-step-ahead predic-\\ntions. These predictions are then recursively fed into the\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\ngenerate simultaneous predictions for multiple pre-deﬁned\\nfuture time steps.\\nMachine learning and deep learning based approaches\\nMachine learning (ML) approaches have shown to improve\\nperformance by addressing high-dimensional and non-linear\\nfeature interactions in a model-free way. These methods in-\\nclude tree-based algorithms, ensemble methods, neural net-\\nwork, autoregression and recurrent neural networks (Hastie,\\nTibshirani, and Friedman 2001). More recent works have ap-', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0}),\n",
       " Document(page_content='work, autoregression and recurrent neural networks (Hastie,\\nTibshirani, and Friedman 2001). More recent works have ap-\\nplied Deep learning (DL) methods on numeric time series\\ndata (Bao, Yue, and Rao 2017; Gensler et al. 2016; Romeu\\net al. 2015; Sagheer and Kotb 2019; Sutskever, Vinyals, and\\nLe 2014). DL automates the process of feature extraction\\nand eliminates the need for domain expertise.\\nSince the introduction of transformers (Vaswani et al.\\n2017), they have become the state of the art model to im-arXiv:2304.04912v1  [cs.LG]  11 Apr 2023', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0}),\n",
       " Document(page_content='Figure 1: Overview of the proposed approach. Quick peak of the transformer encoder architecture on the right (Dosovitskiy\\net al. 2020)\\nprove the performance of NLP applications. The commonly\\nused approach is to pre-train on a large dataset and then ﬁne-\\ntune on a smaller task-speciﬁc dataset (Devlin et al. 2018).\\nTransformers leverage from multi-headed self-attention and\\nreplace the recurrent layers most commonly used in encoder-\\ndecoder architectures. In contrast to RNNs and LSTMs\\nwhere the input data is sequentially processed, transformers\\nbypass the recursion to ingest all inputs at once; thus, trans-\\nformers allow for parallel computations to reduce training\\ntime and do not suffer from long-term memory dependency\\nissues.\\nThe remainder of the paper is organized as follows. We\\ndiscuss our proposed methodology for time series modeling\\nFurther, we discuss our benchmark baseline models along\\nwith the performance evaluation metrics and report the ex-\\nperimental results. Finally, we highlight some concluding re-', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 1}),\n",
       " Document(page_content='with the performance evaluation metrics and report the ex-\\nperimental results. Finally, we highlight some concluding re-\\nmarks and future directions for this study.\\nMethod\\nOur proposed method is called CNN and Transformer based\\ntimeseries modeling (CTTS) as shown in overview Figure 1.\\nPreprocessing\\nWe used standard min-max scaling to standardize each input\\ntime series within [0,1]. Given a raw stock prices time series\\nx, the standardized time series were calculated as\\nxstandardized =x−min(x)\\nmax( x)−min(x)\\nThisxstandardized is then passed into our model to learn the\\nsign of the change in the very next time step, which is de-\\nﬁned as our prediction target.\\nForecasting\\nAs shown in Figure 1, we use 1D CNN kernels to convo-\\nlute through a time series, projecting each local window intoan embedding vector that we call a token. Each token car-\\nries the short-term patterns of the time series. we then add\\npositional embedding (Vaswani et al. 2017) to the token and', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 1}),\n",
       " Document(page_content='ries the short-term patterns of the time series. we then add\\npositional embedding (Vaswani et al. 2017) to the token and\\npass that through a Transformer model to learn the long-term\\ndependencies between these tokens. The transformer model\\noutputs a latent embedding vector of the time series, which\\nis then passed through a Multilayer Perceptron (MLP) with\\nsoftmax activation in the end to generate sign classiﬁcation\\noutputs. The output is in the form of probability for each\\nclass (up, down, ﬂat), where the probability for all 3 classes\\nsum up to 1.\\nExperiments\\nIn our experiments, we benchmarked CTTS - our proposed\\nmethod against 4 methods. 1) DeepAR - a state-of-the-art\\nautoregressive recurrent neural networks-based time series\\nforecasting method, 2) AutoRegressive Integrated Moving\\nAverage (ARIMA), 3) Exponential Moving Average (EMA)\\nand 4) naive constant class predictions. We benchmarked the\\nperformance using sign prediction accuracy and thresholded', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 1}),\n",
       " Document(page_content='and 4) naive constant class predictions. We benchmarked the\\nperformance using sign prediction accuracy and thresholded\\nversion of the sign prediction accuracy (discussed later) us-\\ning our model prediction probabilities. We ran our experi-\\nments on a Linux machine with 8 16GB NVIDIA T4 Ten-\\nsor Core GPUs, and using PyTorch v1.0.0 DL platform in\\nPython 3.6. In all models, we set a ﬁxed random seed for\\nreproducible results.\\nExperimental setup\\nData We used the intraday stock prices of S&P 500 con-\\nstituent stocks obtained from licensed Bloomberg data ser-\\nvice (blo 2022). The data was sampled at 1 minute interval\\nfor the year 2019 (52 weeks, each week has 5 trading days).\\nFor every stock, we sampled 7 time series for each week.\\nData from the ﬁrst three quarters (weeks 1 to 39) were used\\nfor training and validation. Data was randomly split and 80%\\nwas used for training and the remaining 20% for validation.', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 1}),\n",
       " Document(page_content='Method 2-class↑2-class∗↑3-class↑3-class∗↑\\nEMA 53.2% 59.9% 39.5% 41.7%\\nARIMA 50.9% 51.8% 37.5% 38.4%\\nDeepAR 51.1% 53.6% 37.4% 38.7%\\nCTTS 56.7% 66.8% 44.1% 55.2%\\nTable 1: Summary of sign accuracy over the last quarter of\\n2019. Our proposed method CTTS outperforms the base-\\nlines DeepAR, ARIMA and EMA. 2/3-class∗refers to the\\nthresholded version of the sign accuracy. The distribution of\\nsigns in the ground truth test set are as follows: price goes up\\n|down|remains ﬂat: 37.1% |36.5%|26.4%, respectively. If\\nnaively predicting the majority class (up) for all time series,\\nthe sign accuracy will only be 37.1%.\\nWe had around 507K training and 117K validation samples.\\nData from the last quarter (weeks 40 to 52) was used for\\ntesting, totaling 209K time series. For each time series, the\\nﬁrst 80 time steps ( input ) were used to forecast the sign of\\nprice change at the 81st time step ( target ). The overall test\\nset performance is denoted by the aggregate for the evalua-\\ntion metrics over the test set.', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 2}),\n",
       " Document(page_content='price change at the 81st time step ( target ). The overall test\\nset performance is denoted by the aggregate for the evalua-\\ntion metrics over the test set.\\nCTTS In our experiments, we used cross entropy loss, and\\nAdamW (Loshchilov and Hutter 2017) optimizer with batch\\nsize of 64 and max epoch of 100. we used CNN kernel size\\nof 16 and stride 8. Transformer with depth 4 and 4 self-\\nattention heads, with an embedding dimension of 128, and a\\ndrop rate of 0.3 to prevent overﬁtting.\\nBaselines\\nDeepAR DeepAR forecasting algorithm is a supervised\\nlearning algorithm for forecasting 1D time series using au-\\ntoregressive recurrent neural networks. DeepAR beneﬁts\\nfrom training a single model jointly over all of the time se-\\nries. We used a batch size of 128, Adam optimizer, and the\\nnormal distribution loss. The model was trained for a maxi-\\nmum of 300 epochs, with early stopping mechanism set to a\\npatience of 15. The base learning rate was 1e-3, adjusted by\\nlearning rate scheduler with decay factor 0.1 and a patience', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 2}),\n",
       " Document(page_content='patience of 15. The base learning rate was 1e-3, adjusted by\\nlearning rate scheduler with decay factor 0.1 and a patience\\nof 5. We also used dropout with probability 0.1. DeepAR\\ngenerated multiple (200) samples of the prediction target for\\neach time series and we deﬁned the prediction probability\\nover the three classes as the proportion of samples predicted\\nper class.\\nARIMA Autoregressive Integrated Moving Average\\n(ARIMA) models capture autocorrelations in the data\\nusing a combination approach of autoregressive model,\\nmoving average model, and differencing (Wilks 2011). We\\ncompared the continuous valued ARIMA forecasts with the\\nlast known price in the input data to deﬁne the predicted\\nsign, and the corresponding prediction probability was\\ndeﬁned as the percentage of the absolute delta between\\nthe predicted and the last known price with respect to the\\nstandard deviation of the past 80 data points, capped by 1.\\nEMA An exponential moving average (EMA) is a type of', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 2}),\n",
       " Document(page_content='standard deviation of the past 80 data points, capped by 1.\\nEMA An exponential moving average (EMA) is a type of\\nmoving average that places a greater weight and signiﬁcanceon the most recent data points. We used the ”estimated” ini-\\ntialization method, which treats the initial values like param-\\neters, and chooses them to minimize the sum of squared er-\\nrors. Similar to ARIMA, we delta between the predicted and\\nthe last known price to deﬁne the predicted sign, and the\\ncorresponding prediction probability.\\nConstant Class Prediction We tested the three naive\\nbaselines where we always predict a constant sign i.e., ei-\\nther the predicted price always goes up, down, or remains\\nﬂat. The prediction probabilities for these were set to 1 for\\nthe winning class, and 0 for the remaining two classes.\\nMetric\\nWe evaluated all our models on 3-class [class 1-price goes\\nup, class 2-price goes down, class 3-price remains ﬂat] and\\n2-class [class 1-price goes up or remains ﬂat, class 2-price', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 2}),\n",
       " Document(page_content='up, class 2-price goes down, class 3-price remains ﬂat] and\\n2-class [class 1-price goes up or remains ﬂat, class 2-price\\ngoes down]. We used the averaged sign prediction accuracy\\nover all test samples to evaluate our models. Higher accuracy\\nimplies more accurate forecasts.\\nFurther, we evaluate a thresholded version of the sign\\naccuracy, where we deﬁned our threshold as the 75thper-\\ncentile of all predicted probabilities of dominating classes.\\nWe retain only the samples that exceed the threshold and\\ncompute the sign prediction accuracy over these retained\\nsamples.\\nResults & Discussion\\nWe summarize the quantitative benchmark results from the\\nsign accuracy in Table 1. We show the sign accuracy for both\\n2-class and 3-class tasks as explained above. Note that ran-\\ndom guess in 2-class task leads to 50% of accuracy, and\\nrandom guess in 3-class task leads to 33% of accuracy.\\nAs shown in the 2-class and 3-class columns of Table 1,\\nCTTS outperformed all baseline methods in both cases. This', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 2}),\n",
       " Document(page_content='As shown in the 2-class and 3-class columns of Table 1,\\nCTTS outperformed all baseline methods in both cases. This\\ndemonstrates the beneﬁt of combining CNNs and transform-\\ners for time series forecasting.\\nFurther, we evaluate a thresholded version of the sign ac-\\ncuracy, where we deﬁned our threshold as the 75th percentile\\nof all predicted probabilities of dominating classes. We re-\\ntain only the samples that exceed the threshold and compute\\nthe sign prediction accuracy over these retained samples. As\\nshown in 2-class∗and 3-class∗columns of Table 1, the ac-\\ncuracy after thresholding over the prediction probability has\\nincreased for all methods. In addition, the gain of accuracy\\nincrease is the most for our proposed method CTTS. This\\nshows that the class probabilities that CTTS outputs are re-\\nliable. Speciﬁcally, high-conﬁdence predictions from CTTS\\nare often correct, thus thresholding ﬁlters out erroneous low-\\nconﬁdence predictions, leading to a boost in the sign accu-\\nracy.', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 2}),\n",
       " Document(page_content='are often correct, thus thresholding ﬁlters out erroneous low-\\nconﬁdence predictions, leading to a boost in the sign accu-\\nracy.\\nAnother highlight is the signiﬁcant gap that CTTS has\\nachieved for thresholded 3-class∗accuracy compared to\\nother baselines. This can be harnessed in the future for trad-\\ning decision-making. For example, a straightforward trad-\\ning decision can be buy/sell/hold stocks when the predicted\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\ndicted probablitlies are reliable as discussed earlier, the\\namount of stock shares to buy/sell/hold can depend on the', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 2}),\n",
       " Document(page_content='predicted probability, the higher the probability, the more\\nshares to consider.\\nConclusion\\nIn this paper, we tackle the challenging problem of time se-\\nries forecasting of stock prices in the ﬁnancial domain. In\\nthis paper, we demonstrated the combined power of CNN\\nand Transformer to model both short-term and long-term\\ndependencies within a time series. In our experiments over\\nintraday stock price of S&P 500 constituents in year 2019,\\nwe demonstrated the success of the proposed method CTTS\\nin comparison to ARIMA, EMA, and DeepAR, as well as\\nthe potential for using this method for downstream trading\\ndecisions in the future.\\nReferences\\n2022. Bloomberg Market Data.\\nhttps://www.bloomberg.com/professional/product/market-\\ndata/. Accessed: 2022-08-16.\\nBao, W.; Yue, J.; and Rao, Y . 2017. A deep learning frame-\\nwork for ﬁnancial time series using stacked autoencoders\\nand long-short term memory. PloS one , 12(7): e0180944.\\nBrown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.;', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 3}),\n",
       " Document(page_content='and long-short term memory. PloS one , 12(7): e0180944.\\nBrown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.;\\nDhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell,\\nA.; et al. 2020. Language models are few-shot learners. Ad-\\nvances in neural information processing systems , 33: 1877–\\n1901.\\nDevlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.\\nBert: Pre-training of deep bidirectional transformers for lan-\\nguage understanding. arXiv preprint arXiv:1810.04805 .\\nDosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn,\\nD.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.;\\nHeigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16\\nwords: Transformers for image recognition at scale. arXiv\\npreprint arXiv:2010.11929 .\\nGardner Jr, E. S.; and McKenzie, E. 1985. Forecasting\\ntrends in time series. Management Science , 31(10): 1237–\\n1246.\\nGensler, A.; Henze, J.; Sick, B.; and Raabe, N. 2016. Deep\\nLearning for solar power forecasting—An approach using', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 3}),\n",
       " Document(page_content='1246.\\nGensler, A.; Henze, J.; Sick, B.; and Raabe, N. 2016. Deep\\nLearning for solar power forecasting—An approach using\\nAutoEncoder and LSTM Neural Networks. In 2016 IEEE\\ninternational conference on systems, man, and cybernetics\\n(SMC) , 002858–002865. IEEE.\\nHastie, T.; Tibshirani, R.; and Friedman, J. 2001. The el-\\nements of statistical learning. Springer series in statistics.\\nNew York, NY, USA .\\nHochreiter, S.; and Schmidhuber, J. 1997. Long short-term\\nmemory. Neural computation , 9(8): 1735–1780.\\nHolt, C. C. 2004. Forecasting seasonals and trends by expo-\\nnentially weighted moving averages. International journal\\nof forecasting , 20(1): 5–10.\\nLoshchilov, I.; and Hutter, F. 2017. Decoupled weight decay\\nregularization. arXiv preprint arXiv:1711.05101 .\\nMakridakis, S.; Spiliotis, E.; and Assimakopoulos, V . 2020.\\nThe M4 Competition: 100,000 time series and 61 forecasting\\nmethods. International Journal of Forecasting , 36(1): 54–\\n74.Marcellino, M.; Stock, J. H.; and Watson, M. W. 2006. A', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 3}),\n",
       " Document(page_content='methods. International Journal of Forecasting , 36(1): 54–\\n74.Marcellino, M.; Stock, J. H.; and Watson, M. W. 2006. A\\ncomparison of direct and iterated multistep AR methods for\\nforecasting macroeconomic time series. Journal of econo-\\nmetrics , 135(1-2): 499–526.\\nPedersen, L. H. 2019. Efﬁciently inefﬁcient: how smart\\nmoney invests and market prices are determined . Princeton\\nUniversity Press.\\nRen, S.; He, K.; Girshick, R.; and Sun, J. 2015. Faster r-cnn:\\nTowards real-time object detection with region proposal net-\\nworks. Advances in neural information processing systems ,\\n28.\\nRomeu, P.; Zamora-Mart ´ınez, F.; Botella-Rocamora, P.; and\\nPardo, J. 2015. Stacked denoising auto-encoders for short-\\nterm time series forecasting. In Artiﬁcial Neural Networks ,\\n463–486. Springer.\\nRonneberger, O.; Fischer, P.; and Brox, T. 2015. U-net: Con-\\nvolutional networks for biomedical image segmentation. In\\nInternational Conference on Medical image computing and\\ncomputer-assisted intervention , 234–241. Springer.', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 3}),\n",
       " Document(page_content='International Conference on Medical image computing and\\ncomputer-assisted intervention , 234–241. Springer.\\nSagheer, A.; and Kotb, M. 2019. Time series forecasting of\\npetroleum production using deep LSTM recurrent networks.\\nNeurocomputing , 323: 203–213.\\nSalinas, D.; Flunkert, V .; Gasthaus, J.; and Januschowski, T.\\n2020. DeepAR: Probabilistic forecasting with autoregres-\\nsive recurrent networks. International Journal of Forecast-\\ning, 36(3): 1181–1191.\\nSmith, S.; Patwary, M.; Norick, B.; LeGresley, P.; Rajbhan-\\ndari, S.; Casper, J.; Liu, Z.; Prabhumoye, S.; Zerveas, G.;\\nKorthikanti, V .; et al. 2022. Using deepspeed and megatron\\nto train megatron-turing nlg 530b, a large-scale generative\\nlanguage model. arXiv preprint arXiv:2201.11990 .\\nSutskever, I.; Vinyals, O.; and Le, Q. V . 2014. Sequence to\\nsequence learning with neural networks. Advances in neural\\ninformation processing systems , 27.\\nTaieb, S. B.; Sorjamaa, A.; and Bontempi, G. 2010.\\nMultiple-output modeling for multi-step-ahead time series', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 3}),\n",
       " Document(page_content='information processing systems , 27.\\nTaieb, S. B.; Sorjamaa, A.; and Bontempi, G. 2010.\\nMultiple-output modeling for multi-step-ahead time series\\nforecasting. Neurocomputing , 73(10-12): 1950–1957.\\nVaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones,\\nL.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. At-\\ntention is all you need. Advances in neural information pro-\\ncessing systems , 30.\\nWilks, D. S. 2011. Statistical methods in the atmospheric\\nsciences , volume 100. Academic press.\\nWinters, P. R. 1960. Forecasting sales by exponentially\\nweighted moving averages. Management science , 6(3): 324–\\n342.', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 3})]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pdf_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-01-20 01:33:25,676 - WARNING - BigdlNativeEmbeddings has been deprecated, please switch to the new LMEmbeddings API for sepcific models.\n"
     ]
    }
   ],
   "source": [
    "from bigdl.llm.langchain.embeddings import TransformersEmbeddings\n",
    "\n",
    "embeddings = TransformersEmbeddings.from_model_id(model_id=\"/data/vicuna-7b-v1.5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-01-20 00:44:06,382 - INFO - Loading faiss with AVX2 support.\n",
      "2024-01-20 00:44:06,432 - INFO - Successfully loaded faiss with AVX2 support.\n"
     ]
    }
   ],
   "source": [
    "from langchain.vectorstores import FAISS\n",
    "\n",
    "vectorstore_db = FAISS.from_documents(pdf_text, embeddings)\n",
    "retriever = vectorstore_db.as_retriever(search_type=\"similarity\", search_kwargs={\"k\": 6})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts import PromptTemplate\n",
    "custom_prompt_template = \"\"\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\n",
    "If you don't know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\n",
    "Context= {context}\n",
    "History = {history}\n",
    "Question= {question}\n",
    "Helpful Answer:\n",
    "\"\"\"\n",
    "\n",
    "prompt = PromptTemplate(template=custom_prompt_template,\n",
    "                        input_variables=[\"question\", \"context\", \"history\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "issubclass() arg 1 must be a class",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[2], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mchains\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mchat_vector_db\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mprompts\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m QA_PROMPT\n\u001b[1;32m      2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mchains\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mquestion_answering\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m load_qa_chain\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mbigdl\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mllm\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mllms\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m TransformersLLM\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langchain/__init__.py:6\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mimportlib\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m metadata\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtyping\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Optional\n\u001b[0;32m----> 6\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magents\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m MRKLChain, ReActChain, SelfAskWithSearchChain\n\u001b[1;32m      7\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcache\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m BaseCache\n\u001b[1;32m      8\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mchains\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m      9\u001b[0m     ConversationChain,\n\u001b[1;32m     10\u001b[0m     LLMBashChain,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     18\u001b[0m     VectorDBQAWithSourcesChain,\n\u001b[1;32m     19\u001b[0m )\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langchain/agents/__init__.py:2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;124;03m\"\"\"Interface for agents.\"\"\"\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magents\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magent\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m      3\u001b[0m     Agent,\n\u001b[1;32m      4\u001b[0m     AgentExecutor,\n\u001b[1;32m      5\u001b[0m     AgentOutputParser,\n\u001b[1;32m      6\u001b[0m     BaseMultiActionAgent,\n\u001b[1;32m      7\u001b[0m     BaseSingleActionAgent,\n\u001b[1;32m      8\u001b[0m     LLMSingleActionAgent,\n\u001b[1;32m      9\u001b[0m )\n\u001b[1;32m     10\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magents\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magent_iterator\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m AgentExecutorIterator\n\u001b[1;32m     11\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magents\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magent_toolkits\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m     12\u001b[0m     create_csv_agent,\n\u001b[1;32m     13\u001b[0m     create_json_agent,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     23\u001b[0m     create_xorbits_agent,\n\u001b[1;32m     24\u001b[0m )\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langchain/agents/agent.py:15\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01myaml\u001b[39;00m\n\u001b[1;32m     13\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpydantic\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m BaseModel, root_validator\n\u001b[0;32m---> 15\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magents\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magent_iterator\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m AgentExecutorIterator\n\u001b[1;32m     16\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magents\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magent_types\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m AgentType\n\u001b[1;32m     17\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01magents\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtools\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m InvalidTool\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langchain/agents/agent_iterator.py:21\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mfunctools\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m wraps\n\u001b[1;32m      8\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtyping\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m      9\u001b[0m     TYPE_CHECKING,\n\u001b[1;32m     10\u001b[0m     Any,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     18\u001b[0m     Union,\n\u001b[1;32m     19\u001b[0m )\n\u001b[0;32m---> 21\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmanager\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m     22\u001b[0m     AsyncCallbackManager,\n\u001b[1;32m     23\u001b[0m     AsyncCallbackManagerForChainRun,\n\u001b[1;32m     24\u001b[0m     CallbackManager,\n\u001b[1;32m     25\u001b[0m     CallbackManagerForChainRun,\n\u001b[1;32m     26\u001b[0m     Callbacks,\n\u001b[1;32m     27\u001b[0m )\n\u001b[1;32m     28\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mload\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdump\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m dumpd\n\u001b[1;32m     29\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mschema\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m RUN_KEY, AgentAction, AgentFinish, RunInfo\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langchain/callbacks/__init__.py:14\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mhuman\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m HumanApprovalCallbackHandler\n\u001b[1;32m     13\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01minfino_callback\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m InfinoCallbackHandler\n\u001b[0;32m---> 14\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmanager\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m     15\u001b[0m     get_openai_callback,\n\u001b[1;32m     16\u001b[0m     tracing_enabled,\n\u001b[1;32m     17\u001b[0m     tracing_v2_enabled,\n\u001b[1;32m     18\u001b[0m     wandb_tracing_enabled,\n\u001b[1;32m     19\u001b[0m )\n\u001b[1;32m     20\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmlflow_callback\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m MlflowCallbackHandler\n\u001b[1;32m     21\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mopenai_info\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m OpenAICallbackHandler\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langchain/callbacks/manager.py:37\u001b[0m\n\u001b[1;32m     35\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mopenai_info\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m OpenAICallbackHandler\n\u001b[1;32m     36\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mstdout\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m StdOutCallbackHandler\n\u001b[0;32m---> 37\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtracers\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mlangchain\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m LangChainTracer\n\u001b[1;32m     38\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtracers\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mlangchain_v1\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m LangChainTracerV1, TracerSessionV1\n\u001b[1;32m     39\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtracers\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mstdout\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ConsoleCallbackHandler\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langchain/callbacks/tracers/__init__.py:3\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;124;03m\"\"\"Tracers that record execution of LangChain runs.\"\"\"\u001b[39;00m\n\u001b[0;32m----> 3\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtracers\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mlangchain\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m LangChainTracer\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtracers\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mlangchain_v1\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m LangChainTracerV1\n\u001b[1;32m      5\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtracers\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mstdout\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m      6\u001b[0m     ConsoleCallbackHandler,\n\u001b[1;32m      7\u001b[0m     FunctionCallbackHandler,\n\u001b[1;32m      8\u001b[0m )\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langchain/callbacks/tracers/langchain.py:11\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtyping\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Any, Callable, Dict, List, Optional, Set, Union\n\u001b[1;32m      9\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01muuid\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m UUID\n\u001b[0;32m---> 11\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Client\n\u001b[1;32m     13\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtracers\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mbase\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m BaseTracer\n\u001b[1;32m     14\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcallbacks\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtracers\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mschemas\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Run, RunTypeEnum, TracerSession\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langsmith/__init__.py:10\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m metadata\u001b[38;5;241m.\u001b[39mPackageNotFoundError:\n\u001b[1;32m      7\u001b[0m     \u001b[38;5;66;03m# Case where package metadata is not available.\u001b[39;00m\n\u001b[1;32m      8\u001b[0m     __version__ \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m---> 10\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mclient\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Client\n\u001b[1;32m     11\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mevaluation\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mevaluator\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m EvaluationResult, RunEvaluator\n\u001b[1;32m     12\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mrun_helpers\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m trace, traceable\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langsmith/client.py:41\u001b[0m\n\u001b[1;32m     38\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01murllib3\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mutil\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Retry\n\u001b[1;32m     40\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m\n\u001b[0;32m---> 41\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m env \u001b[38;5;28;01mas\u001b[39;00m ls_env\n\u001b[1;32m     42\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m schemas \u001b[38;5;28;01mas\u001b[39;00m ls_schemas\n\u001b[1;32m     43\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m utils \u001b[38;5;28;01mas\u001b[39;00m ls_utils\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langsmith/env/__init__.py:2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;124;03m\"\"\"Utilities to get information about the runtime environment.\"\"\"\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01menv\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01m_runtime_env\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m      3\u001b[0m     get_docker_compose_command,\n\u001b[1;32m      4\u001b[0m     get_docker_compose_version,\n\u001b[1;32m      5\u001b[0m     get_docker_environment,\n\u001b[1;32m      6\u001b[0m     get_docker_version,\n\u001b[1;32m      7\u001b[0m     get_langchain_env_vars,\n\u001b[1;32m      8\u001b[0m     get_langchain_environment,\n\u001b[1;32m      9\u001b[0m     get_release_shas,\n\u001b[1;32m     10\u001b[0m     get_runtime_and_metrics,\n\u001b[1;32m     11\u001b[0m     get_runtime_environment,\n\u001b[1;32m     12\u001b[0m     get_system_metrics,\n\u001b[1;32m     13\u001b[0m )\n\u001b[1;32m     14\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01menv\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01m_git\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m get_git_info\n\u001b[1;32m     16\u001b[0m __all__ \u001b[38;5;241m=\u001b[39m [\n\u001b[1;32m     17\u001b[0m     \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mget_docker_compose_command\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m     18\u001b[0m     \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mget_docker_compose_version\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     27\u001b[0m     \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mget_git_info\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m     28\u001b[0m ]\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langsmith/env/_runtime_env.py:9\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01msubprocess\u001b[39;00m\n\u001b[1;32m      7\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtyping\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Dict, List, Optional, Union\n\u001b[0;32m----> 9\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mutils\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m get_docker_compose_command\n\u001b[1;32m     11\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m     12\u001b[0m     \u001b[38;5;66;03m# psutil is an optional dependency\u001b[39;00m\n\u001b[1;32m     13\u001b[0m     \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mpsutil\u001b[39;00m\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langsmith/utils.py:11\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtyping\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Any, Callable, Dict, List, Mapping, Optional, Tuple, Union\n\u001b[1;32m      9\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mrequests\u001b[39;00m\n\u001b[0;32m---> 11\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangsmith\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m schemas \u001b[38;5;28;01mas\u001b[39;00m ls_schemas\n\u001b[1;32m     13\u001b[0m _LOGGER \u001b[38;5;241m=\u001b[39m logging\u001b[38;5;241m.\u001b[39mgetLogger(\u001b[38;5;18m__name__\u001b[39m)\n\u001b[1;32m     16\u001b[0m \u001b[38;5;28;01mclass\u001b[39;00m \u001b[38;5;21;01mLangSmithError\u001b[39;00m(\u001b[38;5;167;01mException\u001b[39;00m):\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/langsmith/schemas.py:305\u001b[0m\n\u001b[1;32m    301\u001b[0m     \u001b[38;5;28mtype\u001b[39m: \u001b[38;5;28mstr\u001b[39m\n\u001b[1;32m    302\u001b[0m     metadata: Optional[Dict[\u001b[38;5;28mstr\u001b[39m, Any]] \u001b[38;5;241m=\u001b[39m Field(default_factory\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mdict\u001b[39m)\n\u001b[0;32m--> 305\u001b[0m \u001b[38;5;28;01mclass\u001b[39;00m \u001b[38;5;21;01mAPIFeedbackSource\u001b[39;00m(FeedbackSourceBase):\n\u001b[1;32m    306\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"API feedback source.\"\"\"\u001b[39;00m\n\u001b[1;32m    308\u001b[0m     \u001b[38;5;28mtype\u001b[39m: Literal[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mapi\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mapi\u001b[39m\u001b[38;5;124m\"\u001b[39m\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/pydantic/main.py:262\u001b[0m, in \u001b[0;36mpydantic.main.ModelMetaclass.__new__\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/pydantic/fields.py:315\u001b[0m, in \u001b[0;36mpydantic.fields.ModelField.infer\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/pydantic/fields.py:284\u001b[0m, in \u001b[0;36mpydantic.fields.ModelField.__init__\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/pydantic/fields.py:356\u001b[0m, in \u001b[0;36mpydantic.fields.ModelField.prepare\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/pydantic/fields.py:450\u001b[0m, in \u001b[0;36mpydantic.fields.ModelField._type_analysis\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m~/miniconda3/envs/llm-tutorial/lib/python3.9/typing.py:852\u001b[0m, in \u001b[0;36m_SpecialGenericAlias.__subclasscheck__\u001b[0;34m(self, cls)\u001b[0m\n\u001b[1;32m    850\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28missubclass\u001b[39m(\u001b[38;5;28mcls\u001b[39m\u001b[38;5;241m.\u001b[39m__origin__, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__origin__)\n\u001b[1;32m    851\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(\u001b[38;5;28mcls\u001b[39m, _GenericAlias):\n\u001b[0;32m--> 852\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43missubclass\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mcls\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__origin__\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    853\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28msuper\u001b[39m()\u001b[38;5;241m.\u001b[39m\u001b[38;5;21m__subclasscheck__\u001b[39m(\u001b[38;5;28mcls\u001b[39m)\n",
      "\u001b[0;31mTypeError\u001b[0m: issubclass() arg 1 must be a class"
     ]
    }
   ],
   "source": [
    "from langchain.chains.chat_vector_db.prompts import QA_PROMPT\n",
    "from langchain.chains.question_answering import load_qa_chain\n",
    "\n",
    "from bigdl.llm.langchain.llms import TransformersLLM\n",
    "\n",
    "llm = TransformersLLM.from_model_id(\n",
    "        model_id=\"/data/vicuna-7b-v1.5\",\n",
    "        model_kwargs={\"temperature\": 0, \"max_length\": 1024, \"trust_remote_code\": True},\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_chain_with_memory = RetrievalQA.from_chain_type(llm=llm, chain_type='stuff',\n",
    "                                                   retriever = vectorstore_db.as_retriever(),\n",
    "                                                   return_source_documents = True,\n",
    "                                                   chain_type_kwargs = {\"verbose\": True,\n",
    "                                                                        \"prompt\": prompt,\n",
    "                                                                        \"memory\": ConversationBufferMemory(\n",
    "                                                                            input_key=\"question\",\n",
    "                                                                            memory_key=\"history\",\n",
    "                                                                            return_messages=True)})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new StuffDocumentsChain chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3mYou are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\n",
      "If you don't know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\n",
      "Context= and long-term dependencies within a time series, and forecast\n",
      "if the price would go up, down or remain the same (ﬂat) in\n",
      "the future. In our experiments, we demonstrated the success\n",
      "of the proposed method in comparison to commonly adopted\n",
      "statistical and deep learning methods on forecasting intraday\n",
      "stock price change of S&P 500 constituents.\n",
      "Introduction\n",
      "Time series forecasting is challenging, especially in the ﬁ-\n",
      "nancial industry (Pedersen 2019). It involves statistically\n",
      "understanding complex linear and non-linear interactions\n",
      "within historical data to predict the future. In the ﬁnan-\n",
      "cial industry, common applications for forecasting include\n",
      "predicting buy/sell or positive/negative price changes for\n",
      "company stocks traded on the market. Traditional statis-\n",
      "tical approaches commonly adapt linear regression, expo-\n",
      "nential smoothing (Holt 2004; Winters 1960; Gardner Jr\n",
      "and McKenzie 1985) and autoregression models (Makri-\n",
      "dakis, Spiliotis, and Assimakopoulos 2020). With the ad-\n",
      "\n",
      "tistical tools, such as, exponential smoothing (ETS) (Holt\n",
      "2004; Winters 1960; Gardner Jr and McKenzie 1985)\n",
      "and autoregressive integrated moving average (ARIMA)\n",
      "(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\n",
      "merical time series data for making one-step-ahead predic-\n",
      "tions. These predictions are then recursively fed into the\n",
      "future inputs to obtain multi-step forecasts. Multi-horizon\n",
      "forecasting methods such as (Taieb, Sorjamaa, and Bon-\n",
      "tempi 2010; Marcellino, Stock, and Watson 2006) directly\n",
      "generate simultaneous predictions for multiple pre-deﬁned\n",
      "future time steps.\n",
      "Machine learning and deep learning based approaches\n",
      "Machine learning (ML) approaches have shown to improve\n",
      "performance by addressing high-dimensional and non-linear\n",
      "feature interactions in a model-free way. These methods in-\n",
      "clude tree-based algorithms, ensemble methods, neural net-\n",
      "work, autoregression and recurrent neural networks (Hastie,\n",
      "Tibshirani, and Friedman 2001). More recent works have ap-\n",
      "\n",
      "Figure 1: Overview of the proposed approach. Quick peak of the transformer encoder architecture on the right (Dosovitskiy\n",
      "et al. 2020)\n",
      "prove the performance of NLP applications. The commonly\n",
      "used approach is to pre-train on a large dataset and then ﬁne-\n",
      "tune on a smaller task-speciﬁc dataset (Devlin et al. 2018).\n",
      "Transformers leverage from multi-headed self-attention and\n",
      "replace the recurrent layers most commonly used in encoder-\n",
      "decoder architectures. In contrast to RNNs and LSTMs\n",
      "where the input data is sequentially processed, transformers\n",
      "bypass the recursion to ingest all inputs at once; thus, trans-\n",
      "formers allow for parallel computations to reduce training\n",
      "time and do not suffer from long-term memory dependency\n",
      "issues.\n",
      "The remainder of the paper is organized as follows. We\n",
      "discuss our proposed methodology for time series modeling\n",
      "Further, we discuss our benchmark baseline models along\n",
      "with the performance evaluation metrics and report the ex-\n",
      "perimental results. Finally, we highlight some concluding re-\n",
      "\n",
      "are often correct, thus thresholding ﬁlters out erroneous low-\n",
      "conﬁdence predictions, leading to a boost in the sign accu-\n",
      "racy.\n",
      "Another highlight is the signiﬁcant gap that CTTS has\n",
      "achieved for thresholded 3-class∗accuracy compared to\n",
      "other baselines. This can be harnessed in the future for trad-\n",
      "ing decision-making. For example, a straightforward trad-\n",
      "ing decision can be buy/sell/hold stocks when the predicted\n",
      "class is up/down/ﬂat, respectively. Given that CTTS’s pre-\n",
      "dicted probablitlies are reliable as discussed earlier, the\n",
      "amount of stock shares to buy/sell/hold can depend on the\n",
      "History = []\n",
      "Question= what is the main theory of the paper?\n",
      "Helpful Answer:\n",
      "\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/transformers/generation/utils.py:1369: UserWarning: Using `max_length`'s default (4096) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The main theory of the paper is the use of a transformer-based approach for time series forecasting, specifically for intraday stock price change prediction of S&P 500 constituents. The proposed method demonstrates success in comparison to traditional statistical and deep learning methods. The transformer architecture allows for parallel computations and bypasses the recursion issue in RNNs and LSTMs, making it suitable for handling long-term dependencies within a time series.\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'query': 'what is the main theory of the paper?',\n",
       " 'result': \"You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\nIf you don't know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\nContext= and long-term dependencies within a time series, and forecast\\nif the price would go up, down or remain the same (ﬂat) in\\nthe future. In our experiments, we demonstrated the success\\nof the proposed method in comparison to commonly adopted\\nstatistical and deep learning methods on forecasting intraday\\nstock price change of S&P 500 constituents.\\nIntroduction\\nTime series forecasting is challenging, especially in the ﬁ-\\nnancial industry (Pedersen 2019). It involves statistically\\nunderstanding complex linear and non-linear interactions\\nwithin historical data to predict the future. In the ﬁnan-\\ncial industry, common applications for forecasting include\\npredicting buy/sell or positive/negative price changes for\\ncompany stocks traded on the market. Traditional statis-\\ntical approaches commonly adapt linear regression, expo-\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\nand McKenzie 1985) and autoregression models (Makri-\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-\\n\\ntistical tools, such as, exponential smoothing (ETS) (Holt\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\nand autoregressive integrated moving average (ARIMA)\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\nmerical time series data for making one-step-ahead predic-\\ntions. These predictions are then recursively fed into the\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\ngenerate simultaneous predictions for multiple pre-deﬁned\\nfuture time steps.\\nMachine learning and deep learning based approaches\\nMachine learning (ML) approaches have shown to improve\\nperformance by addressing high-dimensional and non-linear\\nfeature interactions in a model-free way. These methods in-\\nclude tree-based algorithms, ensemble methods, neural net-\\nwork, autoregression and recurrent neural networks (Hastie,\\nTibshirani, and Friedman 2001). More recent works have ap-\\n\\nFigure 1: Overview of the proposed approach. Quick peak of the transformer encoder architecture on the right (Dosovitskiy\\net al. 2020)\\nprove the performance of NLP applications. The commonly\\nused approach is to pre-train on a large dataset and then ﬁne-\\ntune on a smaller task-speciﬁc dataset (Devlin et al. 2018).\\nTransformers leverage from multi-headed self-attention and\\nreplace the recurrent layers most commonly used in encoder-\\ndecoder architectures. In contrast to RNNs and LSTMs\\nwhere the input data is sequentially processed, transformers\\nbypass the recursion to ingest all inputs at once; thus, trans-\\nformers allow for parallel computations to reduce training\\ntime and do not suffer from long-term memory dependency\\nissues.\\nThe remainder of the paper is organized as follows. We\\ndiscuss our proposed methodology for time series modeling\\nFurther, we discuss our benchmark baseline models along\\nwith the performance evaluation metrics and report the ex-\\nperimental results. Finally, we highlight some concluding re-\\n\\nare often correct, thus thresholding ﬁlters out erroneous low-\\nconﬁdence predictions, leading to a boost in the sign accu-\\nracy.\\nAnother highlight is the signiﬁcant gap that CTTS has\\nachieved for thresholded 3-class∗accuracy compared to\\nother baselines. This can be harnessed in the future for trad-\\ning decision-making. For example, a straightforward trad-\\ning decision can be buy/sell/hold stocks when the predicted\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\ndicted probablitlies are reliable as discussed earlier, the\\namount of stock shares to buy/sell/hold can depend on the\\nHistory = []\\nQuestion= what is the main theory of the paper?\\nHelpful Answer:\\nThe main theory of the paper is the use of a transformer-based approach for time series forecasting, specifically for intraday stock price change prediction of S&P 500 constituents. The proposed method demonstrates success in comparison to traditional statistical and deep learning methods. The transformer architecture allows for parallel computations and bypasses the recursion issue in RNNs and LSTMs, making it suitable for handling long-term dependencies within a time series.\",\n",
       " 'source_documents': [Document(page_content='and long-term dependencies within a time series, and forecast\\nif the price would go up, down or remain the same (ﬂat) in\\nthe future. In our experiments, we demonstrated the success\\nof the proposed method in comparison to commonly adopted\\nstatistical and deep learning methods on forecasting intraday\\nstock price change of S&P 500 constituents.\\nIntroduction\\nTime series forecasting is challenging, especially in the ﬁ-\\nnancial industry (Pedersen 2019). It involves statistically\\nunderstanding complex linear and non-linear interactions\\nwithin historical data to predict the future. In the ﬁnan-\\ncial industry, common applications for forecasting include\\npredicting buy/sell or positive/negative price changes for\\ncompany stocks traded on the market. Traditional statis-\\ntical approaches commonly adapt linear regression, expo-\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\nand McKenzie 1985) and autoregression models (Makri-\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0}),\n",
       "  Document(page_content='tistical tools, such as, exponential smoothing (ETS) (Holt\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\nand autoregressive integrated moving average (ARIMA)\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\nmerical time series data for making one-step-ahead predic-\\ntions. These predictions are then recursively fed into the\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\ngenerate simultaneous predictions for multiple pre-deﬁned\\nfuture time steps.\\nMachine learning and deep learning based approaches\\nMachine learning (ML) approaches have shown to improve\\nperformance by addressing high-dimensional and non-linear\\nfeature interactions in a model-free way. These methods in-\\nclude tree-based algorithms, ensemble methods, neural net-\\nwork, autoregression and recurrent neural networks (Hastie,\\nTibshirani, and Friedman 2001). More recent works have ap-', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0}),\n",
       "  Document(page_content='Figure 1: Overview of the proposed approach. Quick peak of the transformer encoder architecture on the right (Dosovitskiy\\net al. 2020)\\nprove the performance of NLP applications. The commonly\\nused approach is to pre-train on a large dataset and then ﬁne-\\ntune on a smaller task-speciﬁc dataset (Devlin et al. 2018).\\nTransformers leverage from multi-headed self-attention and\\nreplace the recurrent layers most commonly used in encoder-\\ndecoder architectures. In contrast to RNNs and LSTMs\\nwhere the input data is sequentially processed, transformers\\nbypass the recursion to ingest all inputs at once; thus, trans-\\nformers allow for parallel computations to reduce training\\ntime and do not suffer from long-term memory dependency\\nissues.\\nThe remainder of the paper is organized as follows. We\\ndiscuss our proposed methodology for time series modeling\\nFurther, we discuss our benchmark baseline models along\\nwith the performance evaluation metrics and report the ex-\\nperimental results. Finally, we highlight some concluding re-', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 1}),\n",
       "  Document(page_content='are often correct, thus thresholding ﬁlters out erroneous low-\\nconﬁdence predictions, leading to a boost in the sign accu-\\nracy.\\nAnother highlight is the signiﬁcant gap that CTTS has\\nachieved for thresholded 3-class∗accuracy compared to\\nother baselines. This can be harnessed in the future for trad-\\ning decision-making. For example, a straightforward trad-\\ning decision can be buy/sell/hold stocks when the predicted\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\ndicted probablitlies are reliable as discussed earlier, the\\namount of stock shares to buy/sell/hold can depend on the', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 2})]}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"what is the main theory of the paper?\"\n",
    "qa_chain_with_memory({\"query\": query})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new StuffDocumentsChain chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3mYou are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\n",
      "If you don't know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\n",
      "Context= are often correct, thus thresholding ﬁlters out erroneous low-\n",
      "conﬁdence predictions, leading to a boost in the sign accu-\n",
      "racy.\n",
      "Another highlight is the signiﬁcant gap that CTTS has\n",
      "achieved for thresholded 3-class∗accuracy compared to\n",
      "other baselines. This can be harnessed in the future for trad-\n",
      "ing decision-making. For example, a straightforward trad-\n",
      "ing decision can be buy/sell/hold stocks when the predicted\n",
      "class is up/down/ﬂat, respectively. Given that CTTS’s pre-\n",
      "dicted probablitlies are reliable as discussed earlier, the\n",
      "amount of stock shares to buy/sell/hold can depend on the\n",
      "\n",
      "and 4) naive constant class predictions. We benchmarked the\n",
      "performance using sign prediction accuracy and thresholded\n",
      "version of the sign prediction accuracy (discussed later) us-\n",
      "ing our model prediction probabilities. We ran our experi-\n",
      "ments on a Linux machine with 8 16GB NVIDIA T4 Ten-\n",
      "sor Core GPUs, and using PyTorch v1.0.0 DL platform in\n",
      "Python 3.6. In all models, we set a ﬁxed random seed for\n",
      "reproducible results.\n",
      "Experimental setup\n",
      "Data We used the intraday stock prices of S&P 500 con-\n",
      "stituent stocks obtained from licensed Bloomberg data ser-\n",
      "vice (blo 2022). The data was sampled at 1 minute interval\n",
      "for the year 2019 (52 weeks, each week has 5 trading days).\n",
      "For every stock, we sampled 7 time series for each week.\n",
      "Data from the ﬁrst three quarters (weeks 1 to 39) were used\n",
      "for training and validation. Data was randomly split and 80%\n",
      "was used for training and the remaining 20% for validation.\n",
      "\n",
      "patience of 15. The base learning rate was 1e-3, adjusted by\n",
      "learning rate scheduler with decay factor 0.1 and a patience\n",
      "of 5. We also used dropout with probability 0.1. DeepAR\n",
      "generated multiple (200) samples of the prediction target for\n",
      "each time series and we deﬁned the prediction probability\n",
      "over the three classes as the proportion of samples predicted\n",
      "per class.\n",
      "ARIMA Autoregressive Integrated Moving Average\n",
      "(ARIMA) models capture autocorrelations in the data\n",
      "using a combination approach of autoregressive model,\n",
      "moving average model, and differencing (Wilks 2011). We\n",
      "compared the continuous valued ARIMA forecasts with the\n",
      "last known price in the input data to deﬁne the predicted\n",
      "sign, and the corresponding prediction probability was\n",
      "deﬁned as the percentage of the absolute delta between\n",
      "the predicted and the last known price with respect to the\n",
      "standard deviation of the past 80 data points, capped by 1.\n",
      "EMA An exponential moving average (EMA) is a type of\n",
      "\n",
      "with the performance evaluation metrics and report the ex-\n",
      "perimental results. Finally, we highlight some concluding re-\n",
      "marks and future directions for this study.\n",
      "Method\n",
      "Our proposed method is called CNN and Transformer based\n",
      "timeseries modeling (CTTS) as shown in overview Figure 1.\n",
      "Preprocessing\n",
      "We used standard min-max scaling to standardize each input\n",
      "time series within [0,1]. Given a raw stock prices time series\n",
      "x, the standardized time series were calculated as\n",
      "xstandardized =x−min(x)\n",
      "max( x)−min(x)\n",
      "Thisxstandardized is then passed into our model to learn the\n",
      "sign of the change in the very next time step, which is de-\n",
      "ﬁned as our prediction target.\n",
      "Forecasting\n",
      "As shown in Figure 1, we use 1D CNN kernels to convo-\n",
      "lute through a time series, projecting each local window intoan embedding vector that we call a token. Each token car-\n",
      "ries the short-term patterns of the time series. we then add\n",
      "positional embedding (Vaswani et al. 2017) to the token and\n",
      "History = [HumanMessage(content='what is the main theory of the paper?', additional_kwargs={}, example=False), AIMessage(content=\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\nIf you don't know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\nContext= and long-term dependencies within a time series, and forecast\\nif the price would go up, down or remain the same (ﬂat) in\\nthe future. In our experiments, we demonstrated the success\\nof the proposed method in comparison to commonly adopted\\nstatistical and deep learning methods on forecasting intraday\\nstock price change of S&P 500 constituents.\\nIntroduction\\nTime series forecasting is challenging, especially in the ﬁ-\\nnancial industry (Pedersen 2019). It involves statistically\\nunderstanding complex linear and non-linear interactions\\nwithin historical data to predict the future. In the ﬁnan-\\ncial industry, common applications for forecasting include\\npredicting buy/sell or positive/negative price changes for\\ncompany stocks traded on the market. Traditional statis-\\ntical approaches commonly adapt linear regression, expo-\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\nand McKenzie 1985) and autoregression models (Makri-\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-\\n\\ntistical tools, such as, exponential smoothing (ETS) (Holt\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\nand autoregressive integrated moving average (ARIMA)\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\nmerical time series data for making one-step-ahead predic-\\ntions. These predictions are then recursively fed into the\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\ngenerate simultaneous predictions for multiple pre-deﬁned\\nfuture time steps.\\nMachine learning and deep learning based approaches\\nMachine learning (ML) approaches have shown to improve\\nperformance by addressing high-dimensional and non-linear\\nfeature interactions in a model-free way. These methods in-\\nclude tree-based algorithms, ensemble methods, neural net-\\nwork, autoregression and recurrent neural networks (Hastie,\\nTibshirani, and Friedman 2001). More recent works have ap-\\n\\nFigure 1: Overview of the proposed approach. Quick peak of the transformer encoder architecture on the right (Dosovitskiy\\net al. 2020)\\nprove the performance of NLP applications. The commonly\\nused approach is to pre-train on a large dataset and then ﬁne-\\ntune on a smaller task-speciﬁc dataset (Devlin et al. 2018).\\nTransformers leverage from multi-headed self-attention and\\nreplace the recurrent layers most commonly used in encoder-\\ndecoder architectures. In contrast to RNNs and LSTMs\\nwhere the input data is sequentially processed, transformers\\nbypass the recursion to ingest all inputs at once; thus, trans-\\nformers allow for parallel computations to reduce training\\ntime and do not suffer from long-term memory dependency\\nissues.\\nThe remainder of the paper is organized as follows. We\\ndiscuss our proposed methodology for time series modeling\\nFurther, we discuss our benchmark baseline models along\\nwith the performance evaluation metrics and report the ex-\\nperimental results. Finally, we highlight some concluding re-\\n\\nare often correct, thus thresholding ﬁlters out erroneous low-\\nconﬁdence predictions, leading to a boost in the sign accu-\\nracy.\\nAnother highlight is the signiﬁcant gap that CTTS has\\nachieved for thresholded 3-class∗accuracy compared to\\nother baselines. This can be harnessed in the future for trad-\\ning decision-making. For example, a straightforward trad-\\ning decision can be buy/sell/hold stocks when the predicted\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\ndicted probablitlies are reliable as discussed earlier, the\\namount of stock shares to buy/sell/hold can depend on the\\nHistory = []\\nQuestion= what is the main theory of the paper?\\nHelpful Answer:\\nThe main theory of the paper is the use of a transformer-based approach for time series forecasting, specifically for intraday stock price change prediction of S&P 500 constituents. The proposed method demonstrates success in comparison to traditional statistical and deep learning methods. The transformer architecture allows for parallel computations and bypasses the recursion issue in RNNs and LSTMs, making it suitable for handling long-term dependencies within a time series.\", additional_kwargs={}, example=False)]\n",
      "Question= simply introduce the data sets the paper used.\n",
      "Helpful Answer:\n",
      "\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/transformers/generation/utils.py:1369: UserWarning: Using `max_length`'s default (4096) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The paper used licensed Bloomberg data service for intraday stock prices of S&P 500 constituents for the year 2019, sampled at 1-minute interval for 52 weeks with 7 time series for each week. The data was randomly split into training (80%) and validation (20%) sets.\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'query': 'simply introduce the data sets the paper used.',\n",
       " 'result': 'You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\nIf you don\\'t know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\nContext= are often correct, thus thresholding ﬁlters out erroneous low-\\nconﬁdence predictions, leading to a boost in the sign accu-\\nracy.\\nAnother highlight is the signiﬁcant gap that CTTS has\\nachieved for thresholded 3-class∗accuracy compared to\\nother baselines. This can be harnessed in the future for trad-\\ning decision-making. For example, a straightforward trad-\\ning decision can be buy/sell/hold stocks when the predicted\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\ndicted probablitlies are reliable as discussed earlier, the\\namount of stock shares to buy/sell/hold can depend on the\\n\\nand 4) naive constant class predictions. We benchmarked the\\nperformance using sign prediction accuracy and thresholded\\nversion of the sign prediction accuracy (discussed later) us-\\ning our model prediction probabilities. We ran our experi-\\nments on a Linux machine with 8 16GB NVIDIA T4 Ten-\\nsor Core GPUs, and using PyTorch v1.0.0 DL platform in\\nPython 3.6. In all models, we set a ﬁxed random seed for\\nreproducible results.\\nExperimental setup\\nData We used the intraday stock prices of S&P 500 con-\\nstituent stocks obtained from licensed Bloomberg data ser-\\nvice (blo 2022). The data was sampled at 1 minute interval\\nfor the year 2019 (52 weeks, each week has 5 trading days).\\nFor every stock, we sampled 7 time series for each week.\\nData from the ﬁrst three quarters (weeks 1 to 39) were used\\nfor training and validation. Data was randomly split and 80%\\nwas used for training and the remaining 20% for validation.\\n\\npatience of 15. The base learning rate was 1e-3, adjusted by\\nlearning rate scheduler with decay factor 0.1 and a patience\\nof 5. We also used dropout with probability 0.1. DeepAR\\ngenerated multiple (200) samples of the prediction target for\\neach time series and we deﬁned the prediction probability\\nover the three classes as the proportion of samples predicted\\nper class.\\nARIMA Autoregressive Integrated Moving Average\\n(ARIMA) models capture autocorrelations in the data\\nusing a combination approach of autoregressive model,\\nmoving average model, and differencing (Wilks 2011). We\\ncompared the continuous valued ARIMA forecasts with the\\nlast known price in the input data to deﬁne the predicted\\nsign, and the corresponding prediction probability was\\ndeﬁned as the percentage of the absolute delta between\\nthe predicted and the last known price with respect to the\\nstandard deviation of the past 80 data points, capped by 1.\\nEMA An exponential moving average (EMA) is a type of\\n\\nwith the performance evaluation metrics and report the ex-\\nperimental results. Finally, we highlight some concluding re-\\nmarks and future directions for this study.\\nMethod\\nOur proposed method is called CNN and Transformer based\\ntimeseries modeling (CTTS) as shown in overview Figure 1.\\nPreprocessing\\nWe used standard min-max scaling to standardize each input\\ntime series within [0,1]. Given a raw stock prices time series\\nx, the standardized time series were calculated as\\nxstandardized =x−min(x)\\nmax( x)−min(x)\\nThisxstandardized is then passed into our model to learn the\\nsign of the change in the very next time step, which is de-\\nﬁned as our prediction target.\\nForecasting\\nAs shown in Figure 1, we use 1D CNN kernels to convo-\\nlute through a time series, projecting each local window intoan embedding vector that we call a token. Each token car-\\nries the short-term patterns of the time series. we then add\\npositional embedding (Vaswani et al. 2017) to the token and\\nHistory = [HumanMessage(content=\\'what is the main theory of the paper?\\', additional_kwargs={}, example=False), AIMessage(content=\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\\\nIf you don\\'t know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\\\nContext= and long-term dependencies within a time series, and forecast\\\\nif the price would go up, down or remain the same (ﬂat) in\\\\nthe future. In our experiments, we demonstrated the success\\\\nof the proposed method in comparison to commonly adopted\\\\nstatistical and deep learning methods on forecasting intraday\\\\nstock price change of S&P 500 constituents.\\\\nIntroduction\\\\nTime series forecasting is challenging, especially in the ﬁ-\\\\nnancial industry (Pedersen 2019). It involves statistically\\\\nunderstanding complex linear and non-linear interactions\\\\nwithin historical data to predict the future. In the ﬁnan-\\\\ncial industry, common applications for forecasting include\\\\npredicting buy/sell or positive/negative price changes for\\\\ncompany stocks traded on the market. Traditional statis-\\\\ntical approaches commonly adapt linear regression, expo-\\\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\\\nand McKenzie 1985) and autoregression models (Makri-\\\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-\\\\n\\\\ntistical tools, such as, exponential smoothing (ETS) (Holt\\\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\\\nand autoregressive integrated moving average (ARIMA)\\\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\\\nmerical time series data for making one-step-ahead predic-\\\\ntions. These predictions are then recursively fed into the\\\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\\\ngenerate simultaneous predictions for multiple pre-deﬁned\\\\nfuture time steps.\\\\nMachine learning and deep learning based approaches\\\\nMachine learning (ML) approaches have shown to improve\\\\nperformance by addressing high-dimensional and non-linear\\\\nfeature interactions in a model-free way. These methods in-\\\\nclude tree-based algorithms, ensemble methods, neural net-\\\\nwork, autoregression and recurrent neural networks (Hastie,\\\\nTibshirani, and Friedman 2001). More recent works have ap-\\\\n\\\\nFigure 1: Overview of the proposed approach. Quick peak of the transformer encoder architecture on the right (Dosovitskiy\\\\net al. 2020)\\\\nprove the performance of NLP applications. The commonly\\\\nused approach is to pre-train on a large dataset and then ﬁne-\\\\ntune on a smaller task-speciﬁc dataset (Devlin et al. 2018).\\\\nTransformers leverage from multi-headed self-attention and\\\\nreplace the recurrent layers most commonly used in encoder-\\\\ndecoder architectures. In contrast to RNNs and LSTMs\\\\nwhere the input data is sequentially processed, transformers\\\\nbypass the recursion to ingest all inputs at once; thus, trans-\\\\nformers allow for parallel computations to reduce training\\\\ntime and do not suffer from long-term memory dependency\\\\nissues.\\\\nThe remainder of the paper is organized as follows. We\\\\ndiscuss our proposed methodology for time series modeling\\\\nFurther, we discuss our benchmark baseline models along\\\\nwith the performance evaluation metrics and report the ex-\\\\nperimental results. Finally, we highlight some concluding re-\\\\n\\\\nare often correct, thus thresholding ﬁlters out erroneous low-\\\\nconﬁdence predictions, leading to a boost in the sign accu-\\\\nracy.\\\\nAnother highlight is the signiﬁcant gap that CTTS has\\\\nachieved for thresholded 3-class∗accuracy compared to\\\\nother baselines. This can be harnessed in the future for trad-\\\\ning decision-making. For example, a straightforward trad-\\\\ning decision can be buy/sell/hold stocks when the predicted\\\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\\\ndicted probablitlies are reliable as discussed earlier, the\\\\namount of stock shares to buy/sell/hold can depend on the\\\\nHistory = []\\\\nQuestion= what is the main theory of the paper?\\\\nHelpful Answer:\\\\nThe main theory of the paper is the use of a transformer-based approach for time series forecasting, specifically for intraday stock price change prediction of S&P 500 constituents. The proposed method demonstrates success in comparison to traditional statistical and deep learning methods. The transformer architecture allows for parallel computations and bypasses the recursion issue in RNNs and LSTMs, making it suitable for handling long-term dependencies within a time series.\", additional_kwargs={}, example=False)]\\nQuestion= simply introduce the data sets the paper used.\\nHelpful Answer:\\nThe paper used licensed Bloomberg data service for intraday stock prices of S&P 500 constituents for the year 2019, sampled at 1-minute interval for 52 weeks with 7 time series for each week. The data was randomly split into training (80%) and validation (20%) sets.',\n",
       " 'source_documents': [Document(page_content='are often correct, thus thresholding ﬁlters out erroneous low-\\nconﬁdence predictions, leading to a boost in the sign accu-\\nracy.\\nAnother highlight is the signiﬁcant gap that CTTS has\\nachieved for thresholded 3-class∗accuracy compared to\\nother baselines. This can be harnessed in the future for trad-\\ning decision-making. For example, a straightforward trad-\\ning decision can be buy/sell/hold stocks when the predicted\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\ndicted probablitlies are reliable as discussed earlier, the\\namount of stock shares to buy/sell/hold can depend on the', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 2}),\n",
       "  Document(page_content='and 4) naive constant class predictions. We benchmarked the\\nperformance using sign prediction accuracy and thresholded\\nversion of the sign prediction accuracy (discussed later) us-\\ning our model prediction probabilities. We ran our experi-\\nments on a Linux machine with 8 16GB NVIDIA T4 Ten-\\nsor Core GPUs, and using PyTorch v1.0.0 DL platform in\\nPython 3.6. In all models, we set a ﬁxed random seed for\\nreproducible results.\\nExperimental setup\\nData We used the intraday stock prices of S&P 500 con-\\nstituent stocks obtained from licensed Bloomberg data ser-\\nvice (blo 2022). The data was sampled at 1 minute interval\\nfor the year 2019 (52 weeks, each week has 5 trading days).\\nFor every stock, we sampled 7 time series for each week.\\nData from the ﬁrst three quarters (weeks 1 to 39) were used\\nfor training and validation. Data was randomly split and 80%\\nwas used for training and the remaining 20% for validation.', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 1}),\n",
       "  Document(page_content='patience of 15. The base learning rate was 1e-3, adjusted by\\nlearning rate scheduler with decay factor 0.1 and a patience\\nof 5. We also used dropout with probability 0.1. DeepAR\\ngenerated multiple (200) samples of the prediction target for\\neach time series and we deﬁned the prediction probability\\nover the three classes as the proportion of samples predicted\\nper class.\\nARIMA Autoregressive Integrated Moving Average\\n(ARIMA) models capture autocorrelations in the data\\nusing a combination approach of autoregressive model,\\nmoving average model, and differencing (Wilks 2011). We\\ncompared the continuous valued ARIMA forecasts with the\\nlast known price in the input data to deﬁne the predicted\\nsign, and the corresponding prediction probability was\\ndeﬁned as the percentage of the absolute delta between\\nthe predicted and the last known price with respect to the\\nstandard deviation of the past 80 data points, capped by 1.\\nEMA An exponential moving average (EMA) is a type of', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 2}),\n",
       "  Document(page_content='with the performance evaluation metrics and report the ex-\\nperimental results. Finally, we highlight some concluding re-\\nmarks and future directions for this study.\\nMethod\\nOur proposed method is called CNN and Transformer based\\ntimeseries modeling (CTTS) as shown in overview Figure 1.\\nPreprocessing\\nWe used standard min-max scaling to standardize each input\\ntime series within [0,1]. Given a raw stock prices time series\\nx, the standardized time series were calculated as\\nxstandardized =x−min(x)\\nmax( x)−min(x)\\nThisxstandardized is then passed into our model to learn the\\nsign of the change in the very next time step, which is de-\\nﬁned as our prediction target.\\nForecasting\\nAs shown in Figure 1, we use 1D CNN kernels to convo-\\nlute through a time series, projecting each local window intoan embedding vector that we call a token. Each token car-\\nries the short-term patterns of the time series. we then add\\npositional embedding (Vaswani et al. 2017) to the token and', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 1})]}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"simply introduce the data sets the paper used.\"\n",
    "qa_chain_with_memory({\"query\": query})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4873 > 4096). Running this sequence through the model will result in indexing errors\n",
      "/root/miniconda3/envs/llm-tutorial/lib/python3.9/site-packages/transformers/generation/utils.py:1369: UserWarning: Using `max_length`'s default (4096) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.\n",
      "  warnings.warn(\n",
      "Input length of input_ids is 4873, but `max_length` is set to 4096. This can lead to unexpected behavior. You should consider increasing `max_new_tokens`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new StuffDocumentsChain chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3mYou are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\n",
      "If you don't know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\n",
      "Context= work, autoregression and recurrent neural networks (Hastie,\n",
      "Tibshirani, and Friedman 2001). More recent works have ap-\n",
      "plied Deep learning (DL) methods on numeric time series\n",
      "data (Bao, Yue, and Rao 2017; Gensler et al. 2016; Romeu\n",
      "et al. 2015; Sagheer and Kotb 2019; Sutskever, Vinyals, and\n",
      "Le 2014). DL automates the process of feature extraction\n",
      "and eliminates the need for domain expertise.\n",
      "Since the introduction of transformers (Vaswani et al.\n",
      "2017), they have become the state of the art model to im-arXiv:2304.04912v1  [cs.LG]  11 Apr 2023\n",
      "\n",
      "and long-term dependencies within a time series, and forecast\n",
      "if the price would go up, down or remain the same (ﬂat) in\n",
      "the future. In our experiments, we demonstrated the success\n",
      "of the proposed method in comparison to commonly adopted\n",
      "statistical and deep learning methods on forecasting intraday\n",
      "stock price change of S&P 500 constituents.\n",
      "Introduction\n",
      "Time series forecasting is challenging, especially in the ﬁ-\n",
      "nancial industry (Pedersen 2019). It involves statistically\n",
      "understanding complex linear and non-linear interactions\n",
      "within historical data to predict the future. In the ﬁnan-\n",
      "cial industry, common applications for forecasting include\n",
      "predicting buy/sell or positive/negative price changes for\n",
      "company stocks traded on the market. Traditional statis-\n",
      "tical approaches commonly adapt linear regression, expo-\n",
      "nential smoothing (Holt 2004; Winters 1960; Gardner Jr\n",
      "and McKenzie 1985) and autoregression models (Makri-\n",
      "dakis, Spiliotis, and Assimakopoulos 2020). With the ad-\n",
      "\n",
      "information processing systems , 27.\n",
      "Taieb, S. B.; Sorjamaa, A.; and Bontempi, G. 2010.\n",
      "Multiple-output modeling for multi-step-ahead time series\n",
      "forecasting. Neurocomputing , 73(10-12): 1950–1957.\n",
      "Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones,\n",
      "L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. At-\n",
      "tention is all you need. Advances in neural information pro-\n",
      "cessing systems , 30.\n",
      "Wilks, D. S. 2011. Statistical methods in the atmospheric\n",
      "sciences , volume 100. Academic press.\n",
      "Winters, P. R. 1960. Forecasting sales by exponentially\n",
      "weighted moving averages. Management science , 6(3): 324–\n",
      "342.\n",
      "\n",
      "tistical tools, such as, exponential smoothing (ETS) (Holt\n",
      "2004; Winters 1960; Gardner Jr and McKenzie 1985)\n",
      "and autoregressive integrated moving average (ARIMA)\n",
      "(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\n",
      "merical time series data for making one-step-ahead predic-\n",
      "tions. These predictions are then recursively fed into the\n",
      "future inputs to obtain multi-step forecasts. Multi-horizon\n",
      "forecasting methods such as (Taieb, Sorjamaa, and Bon-\n",
      "tempi 2010; Marcellino, Stock, and Watson 2006) directly\n",
      "generate simultaneous predictions for multiple pre-deﬁned\n",
      "future time steps.\n",
      "Machine learning and deep learning based approaches\n",
      "Machine learning (ML) approaches have shown to improve\n",
      "performance by addressing high-dimensional and non-linear\n",
      "feature interactions in a model-free way. These methods in-\n",
      "clude tree-based algorithms, ensemble methods, neural net-\n",
      "work, autoregression and recurrent neural networks (Hastie,\n",
      "Tibshirani, and Friedman 2001). More recent works have ap-\n",
      "History = [HumanMessage(content='what is the main theory of the paper?', additional_kwargs={}, example=False), AIMessage(content=\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\nIf you don't know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\nContext= and long-term dependencies within a time series, and forecast\\nif the price would go up, down or remain the same (ﬂat) in\\nthe future. In our experiments, we demonstrated the success\\nof the proposed method in comparison to commonly adopted\\nstatistical and deep learning methods on forecasting intraday\\nstock price change of S&P 500 constituents.\\nIntroduction\\nTime series forecasting is challenging, especially in the ﬁ-\\nnancial industry (Pedersen 2019). It involves statistically\\nunderstanding complex linear and non-linear interactions\\nwithin historical data to predict the future. In the ﬁnan-\\ncial industry, common applications for forecasting include\\npredicting buy/sell or positive/negative price changes for\\ncompany stocks traded on the market. Traditional statis-\\ntical approaches commonly adapt linear regression, expo-\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\nand McKenzie 1985) and autoregression models (Makri-\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-\\n\\ntistical tools, such as, exponential smoothing (ETS) (Holt\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\nand autoregressive integrated moving average (ARIMA)\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\nmerical time series data for making one-step-ahead predic-\\ntions. These predictions are then recursively fed into the\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\ngenerate simultaneous predictions for multiple pre-deﬁned\\nfuture time steps.\\nMachine learning and deep learning based approaches\\nMachine learning (ML) approaches have shown to improve\\nperformance by addressing high-dimensional and non-linear\\nfeature interactions in a model-free way. These methods in-\\nclude tree-based algorithms, ensemble methods, neural net-\\nwork, autoregression and recurrent neural networks (Hastie,\\nTibshirani, and Friedman 2001). More recent works have ap-\\n\\nFigure 1: Overview of the proposed approach. Quick peak of the transformer encoder architecture on the right (Dosovitskiy\\net al. 2020)\\nprove the performance of NLP applications. The commonly\\nused approach is to pre-train on a large dataset and then ﬁne-\\ntune on a smaller task-speciﬁc dataset (Devlin et al. 2018).\\nTransformers leverage from multi-headed self-attention and\\nreplace the recurrent layers most commonly used in encoder-\\ndecoder architectures. In contrast to RNNs and LSTMs\\nwhere the input data is sequentially processed, transformers\\nbypass the recursion to ingest all inputs at once; thus, trans-\\nformers allow for parallel computations to reduce training\\ntime and do not suffer from long-term memory dependency\\nissues.\\nThe remainder of the paper is organized as follows. We\\ndiscuss our proposed methodology for time series modeling\\nFurther, we discuss our benchmark baseline models along\\nwith the performance evaluation metrics and report the ex-\\nperimental results. Finally, we highlight some concluding re-\\n\\nare often correct, thus thresholding ﬁlters out erroneous low-\\nconﬁdence predictions, leading to a boost in the sign accu-\\nracy.\\nAnother highlight is the signiﬁcant gap that CTTS has\\nachieved for thresholded 3-class∗accuracy compared to\\nother baselines. This can be harnessed in the future for trad-\\ning decision-making. For example, a straightforward trad-\\ning decision can be buy/sell/hold stocks when the predicted\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\ndicted probablitlies are reliable as discussed earlier, the\\namount of stock shares to buy/sell/hold can depend on the\\nHistory = []\\nQuestion= what is the main theory of the paper?\\nHelpful Answer:\\nThe main theory of the paper is the use of a transformer-based approach for time series forecasting, specifically for intraday stock price change prediction of S&P 500 constituents. The proposed method demonstrates success in comparison to traditional statistical and deep learning methods. The transformer architecture allows for parallel computations and bypasses the recursion issue in RNNs and LSTMs, making it suitable for handling long-term dependencies within a time series.\", additional_kwargs={}, example=False), HumanMessage(content='simply introduce the data sets the paper used.', additional_kwargs={}, example=False), AIMessage(content='You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\nIf you don\\'t know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\nContext= are often correct, thus thresholding ﬁlters out erroneous low-\\nconﬁdence predictions, leading to a boost in the sign accu-\\nracy.\\nAnother highlight is the signiﬁcant gap that CTTS has\\nachieved for thresholded 3-class∗accuracy compared to\\nother baselines. This can be harnessed in the future for trad-\\ning decision-making. For example, a straightforward trad-\\ning decision can be buy/sell/hold stocks when the predicted\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\ndicted probablitlies are reliable as discussed earlier, the\\namount of stock shares to buy/sell/hold can depend on the\\n\\nand 4) naive constant class predictions. We benchmarked the\\nperformance using sign prediction accuracy and thresholded\\nversion of the sign prediction accuracy (discussed later) us-\\ning our model prediction probabilities. We ran our experi-\\nments on a Linux machine with 8 16GB NVIDIA T4 Ten-\\nsor Core GPUs, and using PyTorch v1.0.0 DL platform in\\nPython 3.6. In all models, we set a ﬁxed random seed for\\nreproducible results.\\nExperimental setup\\nData We used the intraday stock prices of S&P 500 con-\\nstituent stocks obtained from licensed Bloomberg data ser-\\nvice (blo 2022). The data was sampled at 1 minute interval\\nfor the year 2019 (52 weeks, each week has 5 trading days).\\nFor every stock, we sampled 7 time series for each week.\\nData from the ﬁrst three quarters (weeks 1 to 39) were used\\nfor training and validation. Data was randomly split and 80%\\nwas used for training and the remaining 20% for validation.\\n\\npatience of 15. The base learning rate was 1e-3, adjusted by\\nlearning rate scheduler with decay factor 0.1 and a patience\\nof 5. We also used dropout with probability 0.1. DeepAR\\ngenerated multiple (200) samples of the prediction target for\\neach time series and we deﬁned the prediction probability\\nover the three classes as the proportion of samples predicted\\nper class.\\nARIMA Autoregressive Integrated Moving Average\\n(ARIMA) models capture autocorrelations in the data\\nusing a combination approach of autoregressive model,\\nmoving average model, and differencing (Wilks 2011). We\\ncompared the continuous valued ARIMA forecasts with the\\nlast known price in the input data to deﬁne the predicted\\nsign, and the corresponding prediction probability was\\ndeﬁned as the percentage of the absolute delta between\\nthe predicted and the last known price with respect to the\\nstandard deviation of the past 80 data points, capped by 1.\\nEMA An exponential moving average (EMA) is a type of\\n\\nwith the performance evaluation metrics and report the ex-\\nperimental results. Finally, we highlight some concluding re-\\nmarks and future directions for this study.\\nMethod\\nOur proposed method is called CNN and Transformer based\\ntimeseries modeling (CTTS) as shown in overview Figure 1.\\nPreprocessing\\nWe used standard min-max scaling to standardize each input\\ntime series within [0,1]. Given a raw stock prices time series\\nx, the standardized time series were calculated as\\nxstandardized =x−min(x)\\nmax( x)−min(x)\\nThisxstandardized is then passed into our model to learn the\\nsign of the change in the very next time step, which is de-\\nﬁned as our prediction target.\\nForecasting\\nAs shown in Figure 1, we use 1D CNN kernels to convo-\\nlute through a time series, projecting each local window intoan embedding vector that we call a token. Each token car-\\nries the short-term patterns of the time series. we then add\\npositional embedding (Vaswani et al. 2017) to the token and\\nHistory = [HumanMessage(content=\\'what is the main theory of the paper?\\', additional_kwargs={}, example=False), AIMessage(content=\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\\\nIf you don\\'t know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\\\nContext= and long-term dependencies within a time series, and forecast\\\\nif the price would go up, down or remain the same (ﬂat) in\\\\nthe future. In our experiments, we demonstrated the success\\\\nof the proposed method in comparison to commonly adopted\\\\nstatistical and deep learning methods on forecasting intraday\\\\nstock price change of S&P 500 constituents.\\\\nIntroduction\\\\nTime series forecasting is challenging, especially in the ﬁ-\\\\nnancial industry (Pedersen 2019). It involves statistically\\\\nunderstanding complex linear and non-linear interactions\\\\nwithin historical data to predict the future. In the ﬁnan-\\\\ncial industry, common applications for forecasting include\\\\npredicting buy/sell or positive/negative price changes for\\\\ncompany stocks traded on the market. Traditional statis-\\\\ntical approaches commonly adapt linear regression, expo-\\\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\\\nand McKenzie 1985) and autoregression models (Makri-\\\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-\\\\n\\\\ntistical tools, such as, exponential smoothing (ETS) (Holt\\\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\\\nand autoregressive integrated moving average (ARIMA)\\\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\\\nmerical time series data for making one-step-ahead predic-\\\\ntions. These predictions are then recursively fed into the\\\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\\\ngenerate simultaneous predictions for multiple pre-deﬁned\\\\nfuture time steps.\\\\nMachine learning and deep learning based approaches\\\\nMachine learning (ML) approaches have shown to improve\\\\nperformance by addressing high-dimensional and non-linear\\\\nfeature interactions in a model-free way. These methods in-\\\\nclude tree-based algorithms, ensemble methods, neural net-\\\\nwork, autoregression and recurrent neural networks (Hastie,\\\\nTibshirani, and Friedman 2001). More recent works have ap-\\\\n\\\\nFigure 1: Overview of the proposed approach. Quick peak of the transformer encoder architecture on the right (Dosovitskiy\\\\net al. 2020)\\\\nprove the performance of NLP applications. The commonly\\\\nused approach is to pre-train on a large dataset and then ﬁne-\\\\ntune on a smaller task-speciﬁc dataset (Devlin et al. 2018).\\\\nTransformers leverage from multi-headed self-attention and\\\\nreplace the recurrent layers most commonly used in encoder-\\\\ndecoder architectures. In contrast to RNNs and LSTMs\\\\nwhere the input data is sequentially processed, transformers\\\\nbypass the recursion to ingest all inputs at once; thus, trans-\\\\nformers allow for parallel computations to reduce training\\\\ntime and do not suffer from long-term memory dependency\\\\nissues.\\\\nThe remainder of the paper is organized as follows. We\\\\ndiscuss our proposed methodology for time series modeling\\\\nFurther, we discuss our benchmark baseline models along\\\\nwith the performance evaluation metrics and report the ex-\\\\nperimental results. Finally, we highlight some concluding re-\\\\n\\\\nare often correct, thus thresholding ﬁlters out erroneous low-\\\\nconﬁdence predictions, leading to a boost in the sign accu-\\\\nracy.\\\\nAnother highlight is the signiﬁcant gap that CTTS has\\\\nachieved for thresholded 3-class∗accuracy compared to\\\\nother baselines. This can be harnessed in the future for trad-\\\\ning decision-making. For example, a straightforward trad-\\\\ning decision can be buy/sell/hold stocks when the predicted\\\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\\\ndicted probablitlies are reliable as discussed earlier, the\\\\namount of stock shares to buy/sell/hold can depend on the\\\\nHistory = []\\\\nQuestion= what is the main theory of the paper?\\\\nHelpful Answer:\\\\nThe main theory of the paper is the use of a transformer-based approach for time series forecasting, specifically for intraday stock price change prediction of S&P 500 constituents. The proposed method demonstrates success in comparison to traditional statistical and deep learning methods. The transformer architecture allows for parallel computations and bypasses the recursion issue in RNNs and LSTMs, making it suitable for handling long-term dependencies within a time series.\", additional_kwargs={}, example=False)]\\nQuestion= simply introduce the data sets the paper used.\\nHelpful Answer:\\nThe paper used licensed Bloomberg data service for intraday stock prices of S&P 500 constituents for the year 2019, sampled at 1-minute interval for 52 weeks with 7 time series for each week. The data was randomly split into training (80%) and validation (20%) sets.', additional_kwargs={}, example=False)]\n",
      "Question= summarize the paper, please.\n",
      "Helpful Answer:\n",
      "\u001b[0m\n",
      "\n",
      "\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'query': 'summarize the paper, please.',\n",
       " 'result': 'You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\nIf you don\\'t know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\nContext= work, autoregression and recurrent neural networks (Hastie,\\nTibshirani, and Friedman 2001). More recent works have ap-\\nplied Deep learning (DL) methods on numeric time series\\ndata (Bao, Yue, and Rao 2017; Gensler et al. 2016; Romeu\\net al. 2015; Sagheer and Kotb 2019; Sutskever, Vinyals, and\\nLe 2014). DL automates the process of feature extraction\\nand eliminates the need for domain expertise.\\nSince the introduction of transformers (Vaswani et al.\\n2017), they have become the state of the art model to im-arXiv:2304.04912v1  [cs.LG]  11 Apr 2023\\n\\nand long-term dependencies within a time series, and forecast\\nif the price would go up, down or remain the same (ﬂat) in\\nthe future. In our experiments, we demonstrated the success\\nof the proposed method in comparison to commonly adopted\\nstatistical and deep learning methods on forecasting intraday\\nstock price change of S&P 500 constituents.\\nIntroduction\\nTime series forecasting is challenging, especially in the ﬁ-\\nnancial industry (Pedersen 2019). It involves statistically\\nunderstanding complex linear and non-linear interactions\\nwithin historical data to predict the future. In the ﬁnan-\\ncial industry, common applications for forecasting include\\npredicting buy/sell or positive/negative price changes for\\ncompany stocks traded on the market. Traditional statis-\\ntical approaches commonly adapt linear regression, expo-\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\nand McKenzie 1985) and autoregression models (Makri-\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-\\n\\ninformation processing systems , 27.\\nTaieb, S. B.; Sorjamaa, A.; and Bontempi, G. 2010.\\nMultiple-output modeling for multi-step-ahead time series\\nforecasting. Neurocomputing , 73(10-12): 1950–1957.\\nVaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones,\\nL.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. At-\\ntention is all you need. Advances in neural information pro-\\ncessing systems , 30.\\nWilks, D. S. 2011. Statistical methods in the atmospheric\\nsciences , volume 100. Academic press.\\nWinters, P. R. 1960. Forecasting sales by exponentially\\nweighted moving averages. Management science , 6(3): 324–\\n342.\\n\\ntistical tools, such as, exponential smoothing (ETS) (Holt\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\nand autoregressive integrated moving average (ARIMA)\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\nmerical time series data for making one-step-ahead predic-\\ntions. These predictions are then recursively fed into the\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\ngenerate simultaneous predictions for multiple pre-deﬁned\\nfuture time steps.\\nMachine learning and deep learning based approaches\\nMachine learning (ML) approaches have shown to improve\\nperformance by addressing high-dimensional and non-linear\\nfeature interactions in a model-free way. These methods in-\\nclude tree-based algorithms, ensemble methods, neural net-\\nwork, autoregression and recurrent neural networks (Hastie,\\nTibshirani, and Friedman 2001). More recent works have ap-\\nHistory = [HumanMessage(content=\\'what is the main theory of the paper?\\', additional_kwargs={}, example=False), AIMessage(content=\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\\\nIf you don\\'t know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\\\nContext= and long-term dependencies within a time series, and forecast\\\\nif the price would go up, down or remain the same (ﬂat) in\\\\nthe future. In our experiments, we demonstrated the success\\\\nof the proposed method in comparison to commonly adopted\\\\nstatistical and deep learning methods on forecasting intraday\\\\nstock price change of S&P 500 constituents.\\\\nIntroduction\\\\nTime series forecasting is challenging, especially in the ﬁ-\\\\nnancial industry (Pedersen 2019). It involves statistically\\\\nunderstanding complex linear and non-linear interactions\\\\nwithin historical data to predict the future. In the ﬁnan-\\\\ncial industry, common applications for forecasting include\\\\npredicting buy/sell or positive/negative price changes for\\\\ncompany stocks traded on the market. Traditional statis-\\\\ntical approaches commonly adapt linear regression, expo-\\\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\\\nand McKenzie 1985) and autoregression models (Makri-\\\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-\\\\n\\\\ntistical tools, such as, exponential smoothing (ETS) (Holt\\\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\\\nand autoregressive integrated moving average (ARIMA)\\\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\\\nmerical time series data for making one-step-ahead predic-\\\\ntions. These predictions are then recursively fed into the\\\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\\\ngenerate simultaneous predictions for multiple pre-deﬁned\\\\nfuture time steps.\\\\nMachine learning and deep learning based approaches\\\\nMachine learning (ML) approaches have shown to improve\\\\nperformance by addressing high-dimensional and non-linear\\\\nfeature interactions in a model-free way. These methods in-\\\\nclude tree-based algorithms, ensemble methods, neural net-\\\\nwork, autoregression and recurrent neural networks (Hastie,\\\\nTibshirani, and Friedman 2001). More recent works have ap-\\\\n\\\\nFigure 1: Overview of the proposed approach. Quick peak of the transformer encoder architecture on the right (Dosovitskiy\\\\net al. 2020)\\\\nprove the performance of NLP applications. The commonly\\\\nused approach is to pre-train on a large dataset and then ﬁne-\\\\ntune on a smaller task-speciﬁc dataset (Devlin et al. 2018).\\\\nTransformers leverage from multi-headed self-attention and\\\\nreplace the recurrent layers most commonly used in encoder-\\\\ndecoder architectures. In contrast to RNNs and LSTMs\\\\nwhere the input data is sequentially processed, transformers\\\\nbypass the recursion to ingest all inputs at once; thus, trans-\\\\nformers allow for parallel computations to reduce training\\\\ntime and do not suffer from long-term memory dependency\\\\nissues.\\\\nThe remainder of the paper is organized as follows. We\\\\ndiscuss our proposed methodology for time series modeling\\\\nFurther, we discuss our benchmark baseline models along\\\\nwith the performance evaluation metrics and report the ex-\\\\nperimental results. Finally, we highlight some concluding re-\\\\n\\\\nare often correct, thus thresholding ﬁlters out erroneous low-\\\\nconﬁdence predictions, leading to a boost in the sign accu-\\\\nracy.\\\\nAnother highlight is the signiﬁcant gap that CTTS has\\\\nachieved for thresholded 3-class∗accuracy compared to\\\\nother baselines. This can be harnessed in the future for trad-\\\\ning decision-making. For example, a straightforward trad-\\\\ning decision can be buy/sell/hold stocks when the predicted\\\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\\\ndicted probablitlies are reliable as discussed earlier, the\\\\namount of stock shares to buy/sell/hold can depend on the\\\\nHistory = []\\\\nQuestion= what is the main theory of the paper?\\\\nHelpful Answer:\\\\nThe main theory of the paper is the use of a transformer-based approach for time series forecasting, specifically for intraday stock price change prediction of S&P 500 constituents. The proposed method demonstrates success in comparison to traditional statistical and deep learning methods. The transformer architecture allows for parallel computations and bypasses the recursion issue in RNNs and LSTMs, making it suitable for handling long-term dependencies within a time series.\", additional_kwargs={}, example=False), HumanMessage(content=\\'simply introduce the data sets the paper used.\\', additional_kwargs={}, example=False), AIMessage(content=\\'You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\\\nIf you don\\\\\\'t know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\\\nContext= are often correct, thus thresholding ﬁlters out erroneous low-\\\\nconﬁdence predictions, leading to a boost in the sign accu-\\\\nracy.\\\\nAnother highlight is the signiﬁcant gap that CTTS has\\\\nachieved for thresholded 3-class∗accuracy compared to\\\\nother baselines. This can be harnessed in the future for trad-\\\\ning decision-making. For example, a straightforward trad-\\\\ning decision can be buy/sell/hold stocks when the predicted\\\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\\\ndicted probablitlies are reliable as discussed earlier, the\\\\namount of stock shares to buy/sell/hold can depend on the\\\\n\\\\nand 4) naive constant class predictions. We benchmarked the\\\\nperformance using sign prediction accuracy and thresholded\\\\nversion of the sign prediction accuracy (discussed later) us-\\\\ning our model prediction probabilities. We ran our experi-\\\\nments on a Linux machine with 8 16GB NVIDIA T4 Ten-\\\\nsor Core GPUs, and using PyTorch v1.0.0 DL platform in\\\\nPython 3.6. In all models, we set a ﬁxed random seed for\\\\nreproducible results.\\\\nExperimental setup\\\\nData We used the intraday stock prices of S&P 500 con-\\\\nstituent stocks obtained from licensed Bloomberg data ser-\\\\nvice (blo 2022). The data was sampled at 1 minute interval\\\\nfor the year 2019 (52 weeks, each week has 5 trading days).\\\\nFor every stock, we sampled 7 time series for each week.\\\\nData from the ﬁrst three quarters (weeks 1 to 39) were used\\\\nfor training and validation. Data was randomly split and 80%\\\\nwas used for training and the remaining 20% for validation.\\\\n\\\\npatience of 15. The base learning rate was 1e-3, adjusted by\\\\nlearning rate scheduler with decay factor 0.1 and a patience\\\\nof 5. We also used dropout with probability 0.1. DeepAR\\\\ngenerated multiple (200) samples of the prediction target for\\\\neach time series and we deﬁned the prediction probability\\\\nover the three classes as the proportion of samples predicted\\\\nper class.\\\\nARIMA Autoregressive Integrated Moving Average\\\\n(ARIMA) models capture autocorrelations in the data\\\\nusing a combination approach of autoregressive model,\\\\nmoving average model, and differencing (Wilks 2011). We\\\\ncompared the continuous valued ARIMA forecasts with the\\\\nlast known price in the input data to deﬁne the predicted\\\\nsign, and the corresponding prediction probability was\\\\ndeﬁned as the percentage of the absolute delta between\\\\nthe predicted and the last known price with respect to the\\\\nstandard deviation of the past 80 data points, capped by 1.\\\\nEMA An exponential moving average (EMA) is a type of\\\\n\\\\nwith the performance evaluation metrics and report the ex-\\\\nperimental results. Finally, we highlight some concluding re-\\\\nmarks and future directions for this study.\\\\nMethod\\\\nOur proposed method is called CNN and Transformer based\\\\ntimeseries modeling (CTTS) as shown in overview Figure 1.\\\\nPreprocessing\\\\nWe used standard min-max scaling to standardize each input\\\\ntime series within [0,1]. Given a raw stock prices time series\\\\nx, the standardized time series were calculated as\\\\nxstandardized =x−min(x)\\\\nmax( x)−min(x)\\\\nThisxstandardized is then passed into our model to learn the\\\\nsign of the change in the very next time step, which is de-\\\\nﬁned as our prediction target.\\\\nForecasting\\\\nAs shown in Figure 1, we use 1D CNN kernels to convo-\\\\nlute through a time series, projecting each local window intoan embedding vector that we call a token. Each token car-\\\\nries the short-term patterns of the time series. we then add\\\\npositional embedding (Vaswani et al. 2017) to the token and\\\\nHistory = [HumanMessage(content=\\\\\\'what is the main theory of the paper?\\\\\\', additional_kwargs={}, example=False), AIMessage(content=\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context and answer the question at the end.\\\\\\\\nIf you don\\\\\\'t know the answer just say you d not know an do not try to make up the answer nor try to use outside souces to answer. Keep the answer as concise as possible.\\\\\\\\nContext= and long-term dependencies within a time series, and forecast\\\\\\\\nif the price would go up, down or remain the same (ﬂat) in\\\\\\\\nthe future. In our experiments, we demonstrated the success\\\\\\\\nof the proposed method in comparison to commonly adopted\\\\\\\\nstatistical and deep learning methods on forecasting intraday\\\\\\\\nstock price change of S&P 500 constituents.\\\\\\\\nIntroduction\\\\\\\\nTime series forecasting is challenging, especially in the ﬁ-\\\\\\\\nnancial industry (Pedersen 2019). It involves statistically\\\\\\\\nunderstanding complex linear and non-linear interactions\\\\\\\\nwithin historical data to predict the future. In the ﬁnan-\\\\\\\\ncial industry, common applications for forecasting include\\\\\\\\npredicting buy/sell or positive/negative price changes for\\\\\\\\ncompany stocks traded on the market. Traditional statis-\\\\\\\\ntical approaches commonly adapt linear regression, expo-\\\\\\\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\\\\\\\nand McKenzie 1985) and autoregression models (Makri-\\\\\\\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-\\\\\\\\n\\\\\\\\ntistical tools, such as, exponential smoothing (ETS) (Holt\\\\\\\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\\\\\\\nand autoregressive integrated moving average (ARIMA)\\\\\\\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\\\\\\\nmerical time series data for making one-step-ahead predic-\\\\\\\\ntions. These predictions are then recursively fed into the\\\\\\\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\\\\\\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\\\\\\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\\\\\\\ngenerate simultaneous predictions for multiple pre-deﬁned\\\\\\\\nfuture time steps.\\\\\\\\nMachine learning and deep learning based approaches\\\\\\\\nMachine learning (ML) approaches have shown to improve\\\\\\\\nperformance by addressing high-dimensional and non-linear\\\\\\\\nfeature interactions in a model-free way. These methods in-\\\\\\\\nclude tree-based algorithms, ensemble methods, neural net-\\\\\\\\nwork, autoregression and recurrent neural networks (Hastie,\\\\\\\\nTibshirani, and Friedman 2001). More recent works have ap-\\\\\\\\n\\\\\\\\nFigure 1: Overview of the proposed approach. Quick peak of the transformer encoder architecture on the right (Dosovitskiy\\\\\\\\net al. 2020)\\\\\\\\nprove the performance of NLP applications. The commonly\\\\\\\\nused approach is to pre-train on a large dataset and then ﬁne-\\\\\\\\ntune on a smaller task-speciﬁc dataset (Devlin et al. 2018).\\\\\\\\nTransformers leverage from multi-headed self-attention and\\\\\\\\nreplace the recurrent layers most commonly used in encoder-\\\\\\\\ndecoder architectures. In contrast to RNNs and LSTMs\\\\\\\\nwhere the input data is sequentially processed, transformers\\\\\\\\nbypass the recursion to ingest all inputs at once; thus, trans-\\\\\\\\nformers allow for parallel computations to reduce training\\\\\\\\ntime and do not suffer from long-term memory dependency\\\\\\\\nissues.\\\\\\\\nThe remainder of the paper is organized as follows. We\\\\\\\\ndiscuss our proposed methodology for time series modeling\\\\\\\\nFurther, we discuss our benchmark baseline models along\\\\\\\\nwith the performance evaluation metrics and report the ex-\\\\\\\\nperimental results. Finally, we highlight some concluding re-\\\\\\\\n\\\\\\\\nare often correct, thus thresholding ﬁlters out erroneous low-\\\\\\\\nconﬁdence predictions, leading to a boost in the sign accu-\\\\\\\\nracy.\\\\\\\\nAnother highlight is the signiﬁcant gap that CTTS has\\\\\\\\nachieved for thresholded 3-class∗accuracy compared to\\\\\\\\nother baselines. This can be harnessed in the future for trad-\\\\\\\\ning decision-making. For example, a straightforward trad-\\\\\\\\ning decision can be buy/sell/hold stocks when the predicted\\\\\\\\nclass is up/down/ﬂat, respectively. Given that CTTS’s pre-\\\\\\\\ndicted probablitlies are reliable as discussed earlier, the\\\\\\\\namount of stock shares to buy/sell/hold can depend on the\\\\\\\\nHistory = []\\\\\\\\nQuestion= what is the main theory of the paper?\\\\\\\\nHelpful Answer:\\\\\\\\nThe main theory of the paper is the use of a transformer-based approach for time series forecasting, specifically for intraday stock price change prediction of S&P 500 constituents. The proposed method demonstrates success in comparison to traditional statistical and deep learning methods. The transformer architecture allows for parallel computations and bypasses the recursion issue in RNNs and LSTMs, making it suitable for handling long-term dependencies within a time series.\", additional_kwargs={}, example=False)]\\\\nQuestion= simply introduce the data sets the paper used.\\\\nHelpful Answer:\\\\nThe paper used licensed Bloomberg data service for intraday stock prices of S&P 500 constituents for the year 2019, sampled at 1-minute interval for 52 weeks with 7 time series for each week. The data was randomly split into training (80%) and validation (20%) sets.\\', additional_kwargs={}, example=False)]\\nQuestion= summarize the paper, please.\\nHelpful Answer:\\n\\n',\n",
       " 'source_documents': [Document(page_content='work, autoregression and recurrent neural networks (Hastie,\\nTibshirani, and Friedman 2001). More recent works have ap-\\nplied Deep learning (DL) methods on numeric time series\\ndata (Bao, Yue, and Rao 2017; Gensler et al. 2016; Romeu\\net al. 2015; Sagheer and Kotb 2019; Sutskever, Vinyals, and\\nLe 2014). DL automates the process of feature extraction\\nand eliminates the need for domain expertise.\\nSince the introduction of transformers (Vaswani et al.\\n2017), they have become the state of the art model to im-arXiv:2304.04912v1  [cs.LG]  11 Apr 2023', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0}),\n",
       "  Document(page_content='and long-term dependencies within a time series, and forecast\\nif the price would go up, down or remain the same (ﬂat) in\\nthe future. In our experiments, we demonstrated the success\\nof the proposed method in comparison to commonly adopted\\nstatistical and deep learning methods on forecasting intraday\\nstock price change of S&P 500 constituents.\\nIntroduction\\nTime series forecasting is challenging, especially in the ﬁ-\\nnancial industry (Pedersen 2019). It involves statistically\\nunderstanding complex linear and non-linear interactions\\nwithin historical data to predict the future. In the ﬁnan-\\ncial industry, common applications for forecasting include\\npredicting buy/sell or positive/negative price changes for\\ncompany stocks traded on the market. Traditional statis-\\ntical approaches commonly adapt linear regression, expo-\\nnential smoothing (Holt 2004; Winters 1960; Gardner Jr\\nand McKenzie 1985) and autoregression models (Makri-\\ndakis, Spiliotis, and Assimakopoulos 2020). With the ad-', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0}),\n",
       "  Document(page_content='information processing systems , 27.\\nTaieb, S. B.; Sorjamaa, A.; and Bontempi, G. 2010.\\nMultiple-output modeling for multi-step-ahead time series\\nforecasting. Neurocomputing , 73(10-12): 1950–1957.\\nVaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones,\\nL.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. At-\\ntention is all you need. Advances in neural information pro-\\ncessing systems , 30.\\nWilks, D. S. 2011. Statistical methods in the atmospheric\\nsciences , volume 100. Academic press.\\nWinters, P. R. 1960. Forecasting sales by exponentially\\nweighted moving averages. Management science , 6(3): 324–\\n342.', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 3}),\n",
       "  Document(page_content='tistical tools, such as, exponential smoothing (ETS) (Holt\\n2004; Winters 1960; Gardner Jr and McKenzie 1985)\\nand autoregressive integrated moving average (ARIMA)\\n(Makridakis, Spiliotis, and Assimakopoulos 2020), on nu-\\nmerical time series data for making one-step-ahead predic-\\ntions. These predictions are then recursively fed into the\\nfuture inputs to obtain multi-step forecasts. Multi-horizon\\nforecasting methods such as (Taieb, Sorjamaa, and Bon-\\ntempi 2010; Marcellino, Stock, and Watson 2006) directly\\ngenerate simultaneous predictions for multiple pre-deﬁned\\nfuture time steps.\\nMachine learning and deep learning based approaches\\nMachine learning (ML) approaches have shown to improve\\nperformance by addressing high-dimensional and non-linear\\nfeature interactions in a model-free way. These methods in-\\nclude tree-based algorithms, ensemble methods, neural net-\\nwork, autoregression and recurrent neural networks (Hastie,\\nTibshirani, and Friedman 2001). More recent works have ap-', metadata={'source': '/data/documents/2304.04912.pdf', 'page': 0})]}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"summarize the paper, please.\"\n",
    "qa_chain_with_memory({\"query\": query})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------number of relevant documents--------------------\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "# query = \"这篇文章的主题是什么\\n\"\n",
    "# docs = docsearch.get_relevant_documents(query)\n",
    "# print(\"-\"*20+\"number of relevant documents\"+\"-\"*20)\n",
    "# print(len(docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm-tutorial",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2

}
